
small (250x250 max)
medium (500x500 max)
Large
Extra Large
large ( > 500x500)
Full Resolution


S. P. Cramer & Associates, Inc. 600 NW Fariss Rd Gresham, OR 97030 ( 503) 491 9577, FAX ( 503) 465 1940 www. spcramer. com MEMO TO: Tim O’Laughlin FROM: Jody Lando, Brian Pyper, Michele Simpson, and Andrea Fuller DATE: May 27, 2005 SUBJECT: Preliminary Review of Statistical Analysis presented in “ Issue 8. River Flows: San Joaquin River at Airport Way Bridge … Comments of the California Department of Fish and Game” The following review addresses primary points of concern pertaining to the California Department of Fish and Game’s ( CDFG) Issue 8 analysis, but is not intended to be exhaustive. Statistical relationships and conclusions can be misleading if drawn without critical review and corroboration. As such, we have attempted to point out weaknesses or concerns that require more rigorous investigation or justification. Given the multitude of statistical concerns raised in this critique and the importance of the related management issues, we conclude that CDFG’s comments and analysis provide an insufficient basis for changing the current Vernalis flow regime. We recommend that the statistical analysis be modified and expanded to address the points presented in this critique. Conclusions that can be defensibly stated should be highlighted and sources of uncertainty should be both minimized and documented. It would also be helpful to establish criteria upon which management decisions can be justified. Without such guidelines, the management decisions appear arbitrary. Overall, biologically justified statistical methods and additional data are needed to provide a reliable analysis upon which to recommend management changes. Defensible identification of the linkages between environmental conditions and biological response requires more robust data, analysis, and assumptions than presented by CDFG. Until that time, it is premature to modify the current management procedures. CDFG’s analyses should be considered highly speculative, given the following: misrepresentation of smolt migration relative to flow in the Vernalis Adaptive Management Plan ( VAMP) period flow management considerations not addressed outside the VAMP limited sample size unsubstantiated survival rate estimates the lack of confounding effects other than flow ( e. g., temperature, fry migration, exports) inference outside the range of the predictive data set reliance on strictly linear relationships without the consideration of density dependence 2 unsupported inclusion of production as a function of flow in compound escapement estimates the use of Sacramento Basin data to estimate adult cohort abundance unconventional calculations of percent increase for various metrics the lack of supporting evidence for smolt survival as a function of flow reflected in the returning adult escapement cohort additional concerns regarding flow projections, data exclusion criteria and effectiveness assumptions Each of the points stated above contributes to the overall uncertainty of the CDFG findings and are described in detail below. ( 1) Misrepresentation of Smolt Migration Relative to Flow in the VAMP Period ( a) In CDFG’s Figure 1, they present the cumulative percent of salmon smolt catch passing Mossdale based on data pooled from 1988 2004. According to the figure, 50% of the salmon smolts migrate outside the VAMP window. However, this general representation is misleading because it does not take into account the extreme variation in migration timing relative to water year type. Flows from 1988 2004 vary widely and influence the migration timing and ultimately the cumulative percentage of smolts passing Mossdale in a given year. For instance, in low water years the cumulative percentage of smolts migrating during the VAMP period is much higher than in high water years. Without consistency in the flow conditions over the period of record, conclusions drawn from Figure 1 are unreliable and the data should not be pooled. ( b) To address the previous point and demonstrate a more reliable analysis of smolt migration during the VAMP period, we calculated and graphed smolt migration accounting for variation in flow ( Figure 1 of this memo). The figure shows the proportion of April May smolt migration occurring in the VAMP Period versus actual flows provided in CDFG’s Table 3 and 6,. By accounting for flow variation, Figure 1 shows that in low water years a greater proportion of smolt migration occurs during the VAMP time period. Consequently, a majority of benefits occur in the VAMP time period during low water years. The proportion of smolt migration observed under actual VAMP flows can then be used to estimate migration benefits resulting from adjusted flows. Table 1 of this memo shows the percent increase in smolt migration that could occur during the pre, post and VAMP time periods under a range of the VAMP flow targets. At the minimum target flow of 3200 cfs, 63% of the benefits are accrued in the VAMP time period, while only 13 and 24% respectively are attributed to pre and post VAMP time periods. This is an important perspective that was excluded from the CDFG analysis. ( 2) Flow Management Considerations Not Addressed Outside the VAMP As part of VAMP, exports are substantially curtailed and held stable during the VAMP period. In the absence of such curtailments, any benefits potentially associated with increased flows would be negated by losses of fish due to exports. Therefore, increasing flows at 3 Vernalis outside of the VAMP timeframe may not result in increased smolt survival. While this is not a statistical issue, it may have substantial bearing on management effectiveness and must be considered when evaluating the benefits of changing current practices. ( 3) Limited Sample Size The key relationship used to calculate possible benefits of flow changes on adult returns is illustrated in CDFG’s Figures 4 and 5, which depict the possible relationship between flow and smolt survival rate to Chipp’s Island from Mossdale. This relationship is based on only five years of data ( 2000 2004). Although it appears that two replicates per year were available, the effective sample size with respect to year to year variability in survival rates is essentially five ( the number of years); replicates within a year could be considered as “ pseudo replicates” and must be treated cautiously. The use of pseudoreplicates is a common statistical pitfall and can lead to erroneous conclusions masked by experimental error. As such, the absence of adequate replicate samples can result in a failure to represent the inherent variability of a system. We recommend a sample size analysis be included to determine the necessary number of true replicates. Without such justification, the accuracy and reliability of the conclusions presented in the CDFG comments are unsupported. ( 4) Unsubstantiated Survival Rate Estimates There is no description of the quality of CDFG’s survival rate estimates. Without such information, survival rate estimates pose another form of uncertainty that could strongly bias the resulting recommendations. ( 5) Lack of Confounding Effects other than Flow Potential confounding effects other than flow ( e. g., temperature, dissolved oxygen, exports, fry contribution) were not included in the calculations and these may account for the observed variability in the estimates. Temperature, dissolved oxygen, downstream flow alterations ( exports), and early fry migration can have significant effects on smolt survival and their exclusion from calculations can lead to an overestimation of the benefits that flow alone can provide to smolt survival. For example, “ the estimated poor survival of juveniles rearing in the Delta in dry and normal water years may be caused by a variety of factors such as predation; entrainment at numerous small, unscreened diversions; unsuitable water quality; and/ or direct mortality at the state and federal pumping facilities in the Delta. Entrainment at the Delta pumping facilities does not appear to occur during very wet years since tagged fry were only collected at the pumping facilities during dry years ( Brandes and McLain 2001).” Also, the contribution of fry to escapement in wet years may artificially inflate the perceived smolt survival with regard to regression of April May flow and escapement 2.5 years later. In order to provide a clearer picture of the relationship of smolt survival as a function of flow management at Vernalis, the effects of fry contribution, temperature, and the downstream flow exports must be investigated. Given the current simplicity of the analysis, the conclusion that more flows provided outside of VAMP will result in large increases in smolt survival can not be substantiated. 4 ( 6) Inference Outside the Range of the Predictive Data Set Due to environmental conditions over the past few years, flows only ranged from approximately 2,500 6,000 during the period of interest so the resulting flow survival relationship is based on a very small range of flows. With so few data, it is unclear if a linear relationship is even valid within the observed flow range and to extend inferences outside the range ( i. e., below 2000 or above 7000 cfs) must be considered as highly speculative. In many years and/ or periods, this highly uncertain flow survival relationship was applied outside of the flow range for which it was derived so survival benefits may be overestimated. In addition, the linear flow vs. survival regression ( CDFG Figures 4 and 5) was forced through zero to avoid survival estimates greater than 100%. This is an arbitrary and potentially misleading procedure which may result in biologically and statistically unsupported conclusions. Manipulating biological data when it doesn’t fit within statistical expectations should be done cautiously and only when there is a biological justification for doing so. Rather than applying a statistical fit without biological justification, alternative methods of statistical analysis would be more appropriate and should be investigated ( e. g., logistic regression assuming binomial data). Biological systems are notoriously complicated to represent statistically, typically requiring much more robust data sets, and rarely do they function within linear confines. ( 7) Reliance on Strictly Linear Relationships without the Consideration of Density Dependence The remaining analyses ( all flow periods, flow intervals, years, benefits and compounding escapement values) were then predicated on the highly speculative flow survival relationship discussed above. In addition, all relationships were assumed to be linear ( density independent) which is probably not the case based on the data. For example, the relationship between Chipps Outmigrants and Cohort Production, for which the linear regression line was forced through zero in CDFG’s Figure 7, indicates a Beverton Holt type relationship ( density dependent) between smolt and adult production. This density dependent relationship indicates that fewer adults are produced per smolt as smolt abundance increases, which in turn implies that as flow increases there is a less than proportional benefit on adult returns ( i. e., survival benefits initially rise as flows increase but eventually level off; and, after this plateau point, additional flows do not provide additional benefits). Similarly, CDFG’s “ compounding escapement” analysis assumed a linear relationship between spawners and smolts, and hence, there is no accounting for density dependence in any of these calculations. A density dependent relationship between spawners and smolts would indicate lower benefits from increased flows than those calculated in CDFG’s analysis, since the relative benefits from flow would decrease as flows increase. ( 8) Unsupported Inclusion of Production as a Function of Flow in Compound Escapement 5 Estimates An additional concern involving CDFG’s “ compounding escapement” pertains to the assumed, but not justified, relationship between flow and production. Until this point, the CDFG analysis considered the relationship of flow to survival. However at this stage, there is an unsubstantiated assertion that the number of smolts at Mossdale ( production) is also a function of flow. So, the compound escapement not only reflects a potential overestimate in density independent survival as a function of flow ( see number 7 above), but also includes an assumption of production benefits. These assertions could dramatically and erroneously inflate compound escapement estimates. ( 9) Use of Sacramento Basin Data to Estimate Adult Cohort Abundance Assignment of returning adults to cohort was based on coded wire tag return data from the Sacramento Basin. It would be more appropriate to use one of the following data sets in decreasing order of preference 1) age data from San Joaquin Basin adult scales in recent years, 2) coded wire tag return data from the San Joaquin basin, and 3) length frequency distribution of adults returning to the San Joaquin Basin ( CDFG Figures 3, 6, and 7, and Table 14). If not corrected, the cohort abundance estimates extrapolated from the Sacramento to the San Joaquin Basin will reflect measurement uncertainty to an undetermined extent. It is possible the uncertainty could mask natural variability or statistical relationships between flow and survival ( 10) Unconventional Calculations of Percent Increase for Various Metrics The CDFG analysis presented percent change for various metrics throughout their report ( CDFG Tables 5, 8, 9, 10, 11 and 15); however, these metrics were based on change relative to the new production level rather than the original production level. The CDFG’s calculation was made as follows: ( P2  P1)/ P2 where P2 is the new production level and P1 is the original production level This is both unconventional and potentially misleading. Standard statistical procedures would calculate the change in production relative to the original production level ( P2  P1)/ P1. The distinction can reflect significantly different results. For example in CDFG’s Table 5, the increases in adult salmon escapement during VAMP was reported to range from 14 59 %  seemingly reasonable values. However, had the percent change been calculated according to the original production level, the reported range would be dramatically larger 16 144%. The 144% magnitude change appears excessive and would have prompted additional scrutiny of the various analyses particularly given the limited data set, the pervasive uncertainty in conclusions, and the weak relationship between flow and adult returns discussed in the following bullet point. This unconventional percent increase calculation was consistently applied throughout the report and masked dramatically large changes resulting from flow modifications. As such, the percent increase calculations should be corrected and detailed justification for the magnitude of change must be provided. 6 ( 11) Lack of Supporting Evidence for Smolt Survival as a Function of Flow Reflected in the Returning Adult Escapement Cohort Given the tentative nature of the relationship between flow and survival postulated in CDFG’s Figure 5, it is informative to examine the relationships between flow, Mossdale smolt estimates, and the returning adult escapement cohort. These data were provided in Table 14 of CDFG’s analysis, and represent 13 years of data ( rather than just the five years that were available for other analyses) with high contrast in average Vernalis flows during the VAMP period ( roughly 1,000 to 20,000 cfs across years). To the extent that the benefits analysis of CDFG’s Figure 5 are accurate, we would expect a strong relationship between flow and adult returns per smolt ( i. e., Mossdale smolt to adult survival rate). These data are plotted in Figure 2 of this memo. Although Figure 2 of this memo shows a similar trend to that presented by CDFG where smolt to adult survival tends to increase with higher flow, the relationship is highly uncertain ( the correlation between the two is not statistically significant) and much weaker and than the extrapolated relationship used by CDFG in their Figure 5. This weaker relationship is evidenced by an R 2 value of 0.12 compared to CDFG’s R 2 value of 0.3787. Note that forcing this relationship through “ zero” as CDFG did is not consistent with the data; an alternative is to fit a nonlinear relationship. Rather than introducing new functional relationships ( i. e., nonlinear relationships) to our analysis at this stage, we used the linear methods set forth by CDFG and calculated a standard regression based on Table 14 data. However, we could not justify forcing the statistical fit through zero in Figures 2 and 3. Despite the weak relationship of smolt to adult survival seen in Figure 2, the most striking aspect of this data is the strong increasing time trend seen across years and this trend is only weakly related to Vernalis flow ( Figure 2). Similar to the smolt to adult survival discussed above, there is only a weak apparent relationship between adult returns per spawner and flows ( Figure 3 of this memo). Note that in contrast to adult per smolt estimates, values of adults per spawner were relatively high during low flow escapement years 1990 1993. The inconsistency of these relationships ( i. e., between adult per smolt and adult per spawner) imply that several critical factors determining adult abundances are not accounted for and that, over the 13 year period examined, there is little evidence of a strong flow survival relationship of the nature used in calculations by CDFG. Table 2 of this memo illustrates the Percent Change in Predicted Subsequent Escapement given adjusted VAMP flow levels using the regression derived from CDFG’s Figure 5 data in contrast to the regression derived from CDFG’s Table 14. Figure 5 data clearly generates a much larger response, particularly during the low flows of the late 1980’ s and early 1990’ s. Although we can not attribute this difference to a specific environmental variable, it does raise concerns regarding the conclusions that CDFG made from the Figure 5 regression equations ( 12) Additional Concerns Regarding Data Exclusion Criteria and Effectiveness Assumptions 7 ( a) Data Exclusion Criteria. CDFG’s Table 9 footnotes that the 1995 smolt migration data point was removed from the escapement analysis. The justification was that > 90% of the smolts outmigrated after May 15 under an average flow of > 20,000 cfs which effectively “ swamped improvements made by much smaller flow increments in other years.” While this data is definitely an outlier compared to most other data points, outliers should not be removed from analyses unless there is a strong biological justification for doing so. Removing outliers such as this may obscure important environmental variability. There were two other data points that could also be considered outliers ( i. e., 1989 and 1998) and selectively removing some outliers while retaining others may introduce bias. For instance, the data for 1989 accounted for approximately 1/ 3 of the benefits attributed to an expanded pre VAMP time period, and greater than 50% of the benefits attributed to an expanded post VAMP time period. The strong effect of 1989’ s singular data point demonstrates a high level of uncertainty in the predictive relationships that were generated by CDFG. ( b) Effectiveness Assumptions. It is premature to evaluate the effectiveness of the current water quality standard as it was not implemented until 1999. Assuming that BY 1998 ( 1999 outmigrants) and since have been affected by implementation of the 1995 WQCP measures, we only have complete datasets for three cohorts assuming that most fish return up to 4 years of age. Three data points are not adequate to account for variability between water year types and to evaluate the potential influence of the current protective measures. 8 Table 1: Percent Increase in Smolt Migration from Adjusted Flow Targets 3200 4450 5700 7000 10000 vamp 63% 66% 66% 66% 68% preVAMP 13% 12% 11% 11% 11% postVAMP 24% 23% 23% 22% 21% total 100% 100% 100% 100% 100% Vernalis Flow Targets Table 2: Percent Change in Predicted Subsequent Escapement Percent Change Year Vernalis Flow Actual VAMP ( Table 3) Predicted Subsequent Escapement ( Table 3) Vernalis Flow Adjusted VAMP ( Table 4) Predicted Subsequent Escapement ( Table 4) Figure 5 regression Regression based on Table 14 1988 2093 2559 3200 3913 35% 6% 1989 2168 9496 3200 14016 32% 6% 1990 1280 362 3200 905 60% 11% 1991 1048 680 3200 2076 67% 13% 1992 1250 371 3200 950 61% 11% 1993 3915 1160 4450 1319 12% 3% 1994 2110 1087 3200 1649 34% 6% 1995 19636 2502 19636 2502 0% 0% 1996 6501 6564 7000 7068 7% 2% 1997 5314 1761 5700 1889 7% 2% 1998 19381 20896 19381 20896 0% 0% 1999 6892 1267 7000 1287 2% 1% 2000 5873 1439 5873 1439 0% 0% 2001 4049 2896 4049 2896 0% 0% 2002 3300 2792 3300 2792 0% 0% 2003 3223 2074 3223 2074 0% 0% 2004 3157 2032 3157 2032 0% 0% 9 Proportion of April May Smolt Migration in VAMP Period 0 . 0 0 . 2 0 . 4 0 . 6 0 . 8 1 . 0 0 5000 10000 15000 20000 25000 Veneralis Flow ( cfs) Proportion Figure 1. Proportion of April May smolt migration occurring in the VAMP Period. 10 0 . 00 0 . 01 0 . 02 0 . 03 0 . 04 0 . 05 0 . 06 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 Escapement Year Adult per smolt 0 5,000 10,000 15,000 20,000 25,000 Vernalis Flow ( cfs) Adult/ Smolt Flow R 2 = 0 . 12 0 . 00 0 . 01 0 . 02 0 . 03 0 . 04 0 . 05 0 . 06 0 5,000 10,000 15,000 20,000 25,000 Vernalis Flow Adult per smolt Figure 2. Top panel: Adult per smolt and Vernalis flow versus escapement year. Bottom panel: Adult per smolt versus Vernalis flow. Note: Data obtained from CDFG’s Table 14. 11 0 1 2 3 4 5 6 7 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 Escapement Year Adult per spawner 0 5,000 10,000 15,000 20,000 25,000 Vernalis Flow ( cfs) Adult per sapwner Flow R 2 = 0 . 10 0 . 0 1 . 0 2 . 0 3 . 0 4 . 0 5 . 0 6 . 0 7 . 0 0 5,000 10,000 15,000 20,000 25,000 Vernalis Flow Adult per spawner Figure 3. Top panel: Adult per spawner and Vernalis flow versus escapement year. Bottom panel: Adult per spawner versus Vernalis flow. Note: Data obtained from CDFG’s Table 14.
Click tabs to swap between content that is broken into logical sections.
Rating  
Title  Preliminary review of statistical analysis presented in "Issue 8. River flows San Joaquin River at Airport Way Bridge...comments of the California Department of Fish and Game" 
Subject  StreamflowControlCaliforniaSan Joaquin River.; Water quality managementCaliforniaSan Joaquin River. 
Description  "May 27, 2005." 
Publisher  S.P. Cramer & Associates 
Contributors  Lando, Jody.; O'Laughlin, Tim. 
Type  Text 
Format  View online 
Identifier  http://www.waterboards.ca.gov/waterrights/water_issues/programs/bay_delta/wq_control_plans/2006wqcp/exhibits/append2/sjrg/sjrg28.pdf 
Language  eng 
Relation  http://worldcat.org/oclc/606652917/viewonline 
DateIssued  2005 
FormatExtent  11 p. ; 166.90 KB. 
RelationRequires  Mode of access: World Wide Web. 
Transcript  S. P. Cramer & Associates, Inc. 600 NW Fariss Rd Gresham, OR 97030 ( 503) 491 9577, FAX ( 503) 465 1940 www. spcramer. com MEMO TO: Tim O’Laughlin FROM: Jody Lando, Brian Pyper, Michele Simpson, and Andrea Fuller DATE: May 27, 2005 SUBJECT: Preliminary Review of Statistical Analysis presented in “ Issue 8. River Flows: San Joaquin River at Airport Way Bridge … Comments of the California Department of Fish and Game” The following review addresses primary points of concern pertaining to the California Department of Fish and Game’s ( CDFG) Issue 8 analysis, but is not intended to be exhaustive. Statistical relationships and conclusions can be misleading if drawn without critical review and corroboration. As such, we have attempted to point out weaknesses or concerns that require more rigorous investigation or justification. Given the multitude of statistical concerns raised in this critique and the importance of the related management issues, we conclude that CDFG’s comments and analysis provide an insufficient basis for changing the current Vernalis flow regime. We recommend that the statistical analysis be modified and expanded to address the points presented in this critique. Conclusions that can be defensibly stated should be highlighted and sources of uncertainty should be both minimized and documented. It would also be helpful to establish criteria upon which management decisions can be justified. Without such guidelines, the management decisions appear arbitrary. Overall, biologically justified statistical methods and additional data are needed to provide a reliable analysis upon which to recommend management changes. Defensible identification of the linkages between environmental conditions and biological response requires more robust data, analysis, and assumptions than presented by CDFG. Until that time, it is premature to modify the current management procedures. CDFG’s analyses should be considered highly speculative, given the following: misrepresentation of smolt migration relative to flow in the Vernalis Adaptive Management Plan ( VAMP) period flow management considerations not addressed outside the VAMP limited sample size unsubstantiated survival rate estimates the lack of confounding effects other than flow ( e. g., temperature, fry migration, exports) inference outside the range of the predictive data set reliance on strictly linear relationships without the consideration of density dependence 2 unsupported inclusion of production as a function of flow in compound escapement estimates the use of Sacramento Basin data to estimate adult cohort abundance unconventional calculations of percent increase for various metrics the lack of supporting evidence for smolt survival as a function of flow reflected in the returning adult escapement cohort additional concerns regarding flow projections, data exclusion criteria and effectiveness assumptions Each of the points stated above contributes to the overall uncertainty of the CDFG findings and are described in detail below. ( 1) Misrepresentation of Smolt Migration Relative to Flow in the VAMP Period ( a) In CDFG’s Figure 1, they present the cumulative percent of salmon smolt catch passing Mossdale based on data pooled from 1988 2004. According to the figure, 50% of the salmon smolts migrate outside the VAMP window. However, this general representation is misleading because it does not take into account the extreme variation in migration timing relative to water year type. Flows from 1988 2004 vary widely and influence the migration timing and ultimately the cumulative percentage of smolts passing Mossdale in a given year. For instance, in low water years the cumulative percentage of smolts migrating during the VAMP period is much higher than in high water years. Without consistency in the flow conditions over the period of record, conclusions drawn from Figure 1 are unreliable and the data should not be pooled. ( b) To address the previous point and demonstrate a more reliable analysis of smolt migration during the VAMP period, we calculated and graphed smolt migration accounting for variation in flow ( Figure 1 of this memo). The figure shows the proportion of April May smolt migration occurring in the VAMP Period versus actual flows provided in CDFG’s Table 3 and 6,. By accounting for flow variation, Figure 1 shows that in low water years a greater proportion of smolt migration occurs during the VAMP time period. Consequently, a majority of benefits occur in the VAMP time period during low water years. The proportion of smolt migration observed under actual VAMP flows can then be used to estimate migration benefits resulting from adjusted flows. Table 1 of this memo shows the percent increase in smolt migration that could occur during the pre, post and VAMP time periods under a range of the VAMP flow targets. At the minimum target flow of 3200 cfs, 63% of the benefits are accrued in the VAMP time period, while only 13 and 24% respectively are attributed to pre and post VAMP time periods. This is an important perspective that was excluded from the CDFG analysis. ( 2) Flow Management Considerations Not Addressed Outside the VAMP As part of VAMP, exports are substantially curtailed and held stable during the VAMP period. In the absence of such curtailments, any benefits potentially associated with increased flows would be negated by losses of fish due to exports. Therefore, increasing flows at 3 Vernalis outside of the VAMP timeframe may not result in increased smolt survival. While this is not a statistical issue, it may have substantial bearing on management effectiveness and must be considered when evaluating the benefits of changing current practices. ( 3) Limited Sample Size The key relationship used to calculate possible benefits of flow changes on adult returns is illustrated in CDFG’s Figures 4 and 5, which depict the possible relationship between flow and smolt survival rate to Chipp’s Island from Mossdale. This relationship is based on only five years of data ( 2000 2004). Although it appears that two replicates per year were available, the effective sample size with respect to year to year variability in survival rates is essentially five ( the number of years); replicates within a year could be considered as “ pseudo replicates” and must be treated cautiously. The use of pseudoreplicates is a common statistical pitfall and can lead to erroneous conclusions masked by experimental error. As such, the absence of adequate replicate samples can result in a failure to represent the inherent variability of a system. We recommend a sample size analysis be included to determine the necessary number of true replicates. Without such justification, the accuracy and reliability of the conclusions presented in the CDFG comments are unsupported. ( 4) Unsubstantiated Survival Rate Estimates There is no description of the quality of CDFG’s survival rate estimates. Without such information, survival rate estimates pose another form of uncertainty that could strongly bias the resulting recommendations. ( 5) Lack of Confounding Effects other than Flow Potential confounding effects other than flow ( e. g., temperature, dissolved oxygen, exports, fry contribution) were not included in the calculations and these may account for the observed variability in the estimates. Temperature, dissolved oxygen, downstream flow alterations ( exports), and early fry migration can have significant effects on smolt survival and their exclusion from calculations can lead to an overestimation of the benefits that flow alone can provide to smolt survival. For example, “ the estimated poor survival of juveniles rearing in the Delta in dry and normal water years may be caused by a variety of factors such as predation; entrainment at numerous small, unscreened diversions; unsuitable water quality; and/ or direct mortality at the state and federal pumping facilities in the Delta. Entrainment at the Delta pumping facilities does not appear to occur during very wet years since tagged fry were only collected at the pumping facilities during dry years ( Brandes and McLain 2001).” Also, the contribution of fry to escapement in wet years may artificially inflate the perceived smolt survival with regard to regression of April May flow and escapement 2.5 years later. In order to provide a clearer picture of the relationship of smolt survival as a function of flow management at Vernalis, the effects of fry contribution, temperature, and the downstream flow exports must be investigated. Given the current simplicity of the analysis, the conclusion that more flows provided outside of VAMP will result in large increases in smolt survival can not be substantiated. 4 ( 6) Inference Outside the Range of the Predictive Data Set Due to environmental conditions over the past few years, flows only ranged from approximately 2,500 6,000 during the period of interest so the resulting flow survival relationship is based on a very small range of flows. With so few data, it is unclear if a linear relationship is even valid within the observed flow range and to extend inferences outside the range ( i. e., below 2000 or above 7000 cfs) must be considered as highly speculative. In many years and/ or periods, this highly uncertain flow survival relationship was applied outside of the flow range for which it was derived so survival benefits may be overestimated. In addition, the linear flow vs. survival regression ( CDFG Figures 4 and 5) was forced through zero to avoid survival estimates greater than 100%. This is an arbitrary and potentially misleading procedure which may result in biologically and statistically unsupported conclusions. Manipulating biological data when it doesn’t fit within statistical expectations should be done cautiously and only when there is a biological justification for doing so. Rather than applying a statistical fit without biological justification, alternative methods of statistical analysis would be more appropriate and should be investigated ( e. g., logistic regression assuming binomial data). Biological systems are notoriously complicated to represent statistically, typically requiring much more robust data sets, and rarely do they function within linear confines. ( 7) Reliance on Strictly Linear Relationships without the Consideration of Density Dependence The remaining analyses ( all flow periods, flow intervals, years, benefits and compounding escapement values) were then predicated on the highly speculative flow survival relationship discussed above. In addition, all relationships were assumed to be linear ( density independent) which is probably not the case based on the data. For example, the relationship between Chipps Outmigrants and Cohort Production, for which the linear regression line was forced through zero in CDFG’s Figure 7, indicates a Beverton Holt type relationship ( density dependent) between smolt and adult production. This density dependent relationship indicates that fewer adults are produced per smolt as smolt abundance increases, which in turn implies that as flow increases there is a less than proportional benefit on adult returns ( i. e., survival benefits initially rise as flows increase but eventually level off; and, after this plateau point, additional flows do not provide additional benefits). Similarly, CDFG’s “ compounding escapement” analysis assumed a linear relationship between spawners and smolts, and hence, there is no accounting for density dependence in any of these calculations. A density dependent relationship between spawners and smolts would indicate lower benefits from increased flows than those calculated in CDFG’s analysis, since the relative benefits from flow would decrease as flows increase. ( 8) Unsupported Inclusion of Production as a Function of Flow in Compound Escapement 5 Estimates An additional concern involving CDFG’s “ compounding escapement” pertains to the assumed, but not justified, relationship between flow and production. Until this point, the CDFG analysis considered the relationship of flow to survival. However at this stage, there is an unsubstantiated assertion that the number of smolts at Mossdale ( production) is also a function of flow. So, the compound escapement not only reflects a potential overestimate in density independent survival as a function of flow ( see number 7 above), but also includes an assumption of production benefits. These assertions could dramatically and erroneously inflate compound escapement estimates. ( 9) Use of Sacramento Basin Data to Estimate Adult Cohort Abundance Assignment of returning adults to cohort was based on coded wire tag return data from the Sacramento Basin. It would be more appropriate to use one of the following data sets in decreasing order of preference 1) age data from San Joaquin Basin adult scales in recent years, 2) coded wire tag return data from the San Joaquin basin, and 3) length frequency distribution of adults returning to the San Joaquin Basin ( CDFG Figures 3, 6, and 7, and Table 14). If not corrected, the cohort abundance estimates extrapolated from the Sacramento to the San Joaquin Basin will reflect measurement uncertainty to an undetermined extent. It is possible the uncertainty could mask natural variability or statistical relationships between flow and survival ( 10) Unconventional Calculations of Percent Increase for Various Metrics The CDFG analysis presented percent change for various metrics throughout their report ( CDFG Tables 5, 8, 9, 10, 11 and 15); however, these metrics were based on change relative to the new production level rather than the original production level. The CDFG’s calculation was made as follows: ( P2  P1)/ P2 where P2 is the new production level and P1 is the original production level This is both unconventional and potentially misleading. Standard statistical procedures would calculate the change in production relative to the original production level ( P2  P1)/ P1. The distinction can reflect significantly different results. For example in CDFG’s Table 5, the increases in adult salmon escapement during VAMP was reported to range from 14 59 %  seemingly reasonable values. However, had the percent change been calculated according to the original production level, the reported range would be dramatically larger 16 144%. The 144% magnitude change appears excessive and would have prompted additional scrutiny of the various analyses particularly given the limited data set, the pervasive uncertainty in conclusions, and the weak relationship between flow and adult returns discussed in the following bullet point. This unconventional percent increase calculation was consistently applied throughout the report and masked dramatically large changes resulting from flow modifications. As such, the percent increase calculations should be corrected and detailed justification for the magnitude of change must be provided. 6 ( 11) Lack of Supporting Evidence for Smolt Survival as a Function of Flow Reflected in the Returning Adult Escapement Cohort Given the tentative nature of the relationship between flow and survival postulated in CDFG’s Figure 5, it is informative to examine the relationships between flow, Mossdale smolt estimates, and the returning adult escapement cohort. These data were provided in Table 14 of CDFG’s analysis, and represent 13 years of data ( rather than just the five years that were available for other analyses) with high contrast in average Vernalis flows during the VAMP period ( roughly 1,000 to 20,000 cfs across years). To the extent that the benefits analysis of CDFG’s Figure 5 are accurate, we would expect a strong relationship between flow and adult returns per smolt ( i. e., Mossdale smolt to adult survival rate). These data are plotted in Figure 2 of this memo. Although Figure 2 of this memo shows a similar trend to that presented by CDFG where smolt to adult survival tends to increase with higher flow, the relationship is highly uncertain ( the correlation between the two is not statistically significant) and much weaker and than the extrapolated relationship used by CDFG in their Figure 5. This weaker relationship is evidenced by an R 2 value of 0.12 compared to CDFG’s R 2 value of 0.3787. Note that forcing this relationship through “ zero” as CDFG did is not consistent with the data; an alternative is to fit a nonlinear relationship. Rather than introducing new functional relationships ( i. e., nonlinear relationships) to our analysis at this stage, we used the linear methods set forth by CDFG and calculated a standard regression based on Table 14 data. However, we could not justify forcing the statistical fit through zero in Figures 2 and 3. Despite the weak relationship of smolt to adult survival seen in Figure 2, the most striking aspect of this data is the strong increasing time trend seen across years and this trend is only weakly related to Vernalis flow ( Figure 2). Similar to the smolt to adult survival discussed above, there is only a weak apparent relationship between adult returns per spawner and flows ( Figure 3 of this memo). Note that in contrast to adult per smolt estimates, values of adults per spawner were relatively high during low flow escapement years 1990 1993. The inconsistency of these relationships ( i. e., between adult per smolt and adult per spawner) imply that several critical factors determining adult abundances are not accounted for and that, over the 13 year period examined, there is little evidence of a strong flow survival relationship of the nature used in calculations by CDFG. Table 2 of this memo illustrates the Percent Change in Predicted Subsequent Escapement given adjusted VAMP flow levels using the regression derived from CDFG’s Figure 5 data in contrast to the regression derived from CDFG’s Table 14. Figure 5 data clearly generates a much larger response, particularly during the low flows of the late 1980’ s and early 1990’ s. Although we can not attribute this difference to a specific environmental variable, it does raise concerns regarding the conclusions that CDFG made from the Figure 5 regression equations ( 12) Additional Concerns Regarding Data Exclusion Criteria and Effectiveness Assumptions 7 ( a) Data Exclusion Criteria. CDFG’s Table 9 footnotes that the 1995 smolt migration data point was removed from the escapement analysis. The justification was that > 90% of the smolts outmigrated after May 15 under an average flow of > 20,000 cfs which effectively “ swamped improvements made by much smaller flow increments in other years.” While this data is definitely an outlier compared to most other data points, outliers should not be removed from analyses unless there is a strong biological justification for doing so. Removing outliers such as this may obscure important environmental variability. There were two other data points that could also be considered outliers ( i. e., 1989 and 1998) and selectively removing some outliers while retaining others may introduce bias. For instance, the data for 1989 accounted for approximately 1/ 3 of the benefits attributed to an expanded pre VAMP time period, and greater than 50% of the benefits attributed to an expanded post VAMP time period. The strong effect of 1989’ s singular data point demonstrates a high level of uncertainty in the predictive relationships that were generated by CDFG. ( b) Effectiveness Assumptions. It is premature to evaluate the effectiveness of the current water quality standard as it was not implemented until 1999. Assuming that BY 1998 ( 1999 outmigrants) and since have been affected by implementation of the 1995 WQCP measures, we only have complete datasets for three cohorts assuming that most fish return up to 4 years of age. Three data points are not adequate to account for variability between water year types and to evaluate the potential influence of the current protective measures. 8 Table 1: Percent Increase in Smolt Migration from Adjusted Flow Targets 3200 4450 5700 7000 10000 vamp 63% 66% 66% 66% 68% preVAMP 13% 12% 11% 11% 11% postVAMP 24% 23% 23% 22% 21% total 100% 100% 100% 100% 100% Vernalis Flow Targets Table 2: Percent Change in Predicted Subsequent Escapement Percent Change Year Vernalis Flow Actual VAMP ( Table 3) Predicted Subsequent Escapement ( Table 3) Vernalis Flow Adjusted VAMP ( Table 4) Predicted Subsequent Escapement ( Table 4) Figure 5 regression Regression based on Table 14 1988 2093 2559 3200 3913 35% 6% 1989 2168 9496 3200 14016 32% 6% 1990 1280 362 3200 905 60% 11% 1991 1048 680 3200 2076 67% 13% 1992 1250 371 3200 950 61% 11% 1993 3915 1160 4450 1319 12% 3% 1994 2110 1087 3200 1649 34% 6% 1995 19636 2502 19636 2502 0% 0% 1996 6501 6564 7000 7068 7% 2% 1997 5314 1761 5700 1889 7% 2% 1998 19381 20896 19381 20896 0% 0% 1999 6892 1267 7000 1287 2% 1% 2000 5873 1439 5873 1439 0% 0% 2001 4049 2896 4049 2896 0% 0% 2002 3300 2792 3300 2792 0% 0% 2003 3223 2074 3223 2074 0% 0% 2004 3157 2032 3157 2032 0% 0% 9 Proportion of April May Smolt Migration in VAMP Period 0 . 0 0 . 2 0 . 4 0 . 6 0 . 8 1 . 0 0 5000 10000 15000 20000 25000 Veneralis Flow ( cfs) Proportion Figure 1. Proportion of April May smolt migration occurring in the VAMP Period. 10 0 . 00 0 . 01 0 . 02 0 . 03 0 . 04 0 . 05 0 . 06 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 Escapement Year Adult per smolt 0 5,000 10,000 15,000 20,000 25,000 Vernalis Flow ( cfs) Adult/ Smolt Flow R 2 = 0 . 12 0 . 00 0 . 01 0 . 02 0 . 03 0 . 04 0 . 05 0 . 06 0 5,000 10,000 15,000 20,000 25,000 Vernalis Flow Adult per smolt Figure 2. Top panel: Adult per smolt and Vernalis flow versus escapement year. Bottom panel: Adult per smolt versus Vernalis flow. Note: Data obtained from CDFG’s Table 14. 11 0 1 2 3 4 5 6 7 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 Escapement Year Adult per spawner 0 5,000 10,000 15,000 20,000 25,000 Vernalis Flow ( cfs) Adult per sapwner Flow R 2 = 0 . 10 0 . 0 1 . 0 2 . 0 3 . 0 4 . 0 5 . 0 6 . 0 7 . 0 0 5,000 10,000 15,000 20,000 25,000 Vernalis Flow Adult per spawner Figure 3. Top panel: Adult per spawner and Vernalis flow versus escapement year. Bottom panel: Adult per spawner versus Vernalis flow. Note: Data obtained from CDFG’s Table 14. 
OCLC number  606652917 



W 


