Ralston et al.: A meta-analytic approach to quantifying scientific uncertainty in stock assessments 
221 
residuals and calculate the standard deviation of the 
pooled set. The first method gives each species equal 
weight and does not overemphasize stocks that have 
been assessed many times (e.g., Pacific whiting). Con- 
versely, the second method treats each data point as 
an independent observation. Neither approach is ideal 
given the lack of independence in the data. 
Results 
Most of the groundfish and coastal pelagic species stock 
assessments that have been conducted for the PFMC 
have employed the stock synthesis framework (Methot, 
2000), which provides a very flexible, integrated model- 
ing environment. For our analysis we considered only 
data-rich stocks that have been assessed more than 
once (15 groundfish and two coastal pelagic species 
stocks) — an approach that excluded many species from 
consideration. Owing to the large number of citations 
(81) needed to fully document the assessment literature 
of these stocks, all of which appear in the stock assess- 
ment and fishery evaluation documents produced by the 
PFMC, we present only summary information for each 
assessment that includes the stock, year, and author- 
ship (Table l) 4 . 
There is a preponderance of rockfish among the 
17 species analyzed. The number of assessments in- 
cluded in the meta-analysis ranged from two (chili- 
pepper [Sebastes goodei ]) to a high of fifteen (Pacific 
whiting) (Table 2). Results presented in Figure 2 (A 
and B) show biomass trajectories from 1970 through 
2009 for the 16 stocks that were not whiting stocks. 
Note that there is good correspondence among assess- 
ments for some species (e.g., darkblotched rockfish 
[Sebastes crameri ]) and poor correspondence for oth- 
ers (e.g., shortspine thornyhead [Sebastolobus alas- 
canus ]). One should be cautious in interpreting this 
correspondence to indicate the degree of uncertainty 
in stock biomass because random variation in corre- 
spondence between species is to be expected. In the 
case of shortspine thornyhead, new information indi- 
cating dome-shaped selectivity was largely responsible 
for the large change in the biomass estimates for the 
2005 assessment. 
Comparisons of methods 
When the assessment data were restricted to the last 
twenty years, the three approaches (all ratio combina- 
tions, deviations from the mean, and deviations from 
the most recent assessment) yielded average estimates 
of o over all stocks equal to 0.382, 0.337, and 0.307, 
respectively. Approach two (i.e., squared deviations from 
4 Individuals should contact the PFMC ( John.DeVore@noaa. 
gov) or visit http://www.pcoimcil.org/groundfish/stock-assess- 
ments/ or http://www.pcouncil.org/coastal-pelagic-species/ 
stock-assessment-and-fishery-evaluation-safe -documents/ 
for copies of specific assessment documents. 
the mean in log-space) was selected as the preferred 
method for calculating uncertainty by the Scientific 
and Statistical Committee because it had two desirable 
features, i.e., deviations were calculated from the best 
estimate of central tendency and estimated values of a 
were unlikely to change markedly with new assessments 
(unlike approach three, which relies on the most recent 
assessment as the reference). Coincidently the calcula- 
tion produced an intermediate result among the three 
approaches. 
Similarly, a sensitivity test of the results to the num- 
ber of years included in the calculation revealed that 
estimates of a were robust to the time period used in the 
calculation. For example, when only the last 15 years of 
data were used, o was 0.338 (compared with 0.337 for 20 
years). Likewise, when the final 10 and 5 years of data 
were used, estimates of a were 0.371 and 0.344, respec- 
tively. Note that in these latter two cases some species 
were excluded because of sparseness of data for these 
species. Hence, a standard temporal window equal to the 
last 20 years of assessment data was adopted as the ba- 
sis for quantifying variation among stock assessments. 
Stock-specific results 
Figure 3 shows the distributions of residuals for the 17 
stocks based on the selected approach. Note that some 
species (e.g., chilipepper and shortspine thornyhead) 
show a strongly bimodal distribution — a pattern that 
results when few assessments are available and bio- 
mass trajectories do not intersect. However, most of the 
distributions are unimodal, generally symmetric, and 
centered on or near zero. 
Table 2 presents the number of deviations and the 
estimated log-scale standard deviation (cr) for each of 
the stocks, which collectively ranged from 0.103 (dark- 
blotched rockfish) to 0.923 (shortspine thornyhead) with 
an average of 0.337. Also presented in the table are 
the estimated asymptotic coefficients of variation (CVs) 
for terminal biomass from the most recently complet- 
ed stock assessment. These CVs, which approximate 
within-assessment estimation error, ranged from 9% 
(shortspine thornyhead and Dover sole [Microstomus 
pacificus ]) to 41% (Pacific sardine [Sardmops sagax]), 
with a mean of 18%. This is undoubtedly an underes- 
timate, however, because of the presence of key fixed 
parameters (e.g., natural mortality) in almost all the 
assessments we reviewed. 
To compare among-assessment variation to within- 
assessment variation, the log-scale standard deviation 
estimates were expressed as CVs on the arithmetic 
scale (Johnson and Kotz, 1970), and the two statistics 
were plotted against one another (Figure 4). It is evi- 
dent that shortspine thornyhead is an outlier, with the 
lowest “within” CV (stochastic uncertainty) and the 
highest “among” CV (inductive uncertainty). As a rule, 
among-assessment CVs (mean=36%) were greater than 
within-assessment CVs, as evidenced by the preponder- 
ance of points falling to the right side of the line of 
equality. 
