156 
Fishery Bulletin 119(2-3) 
et al., 1998; Gabriel and Mace, 1999; Methot et al., 2014). 
The estimated values of F/F};,,;, and SSB/SSB;,,,;, from the 
EMs were compared with the true values from the OM to 
verify whether the estimated stock status (i.e., overfished 
or overfishing) matched the true status. 
The bias and variability of the bias of the EMs were deter- 
mined by calculating relative error (RE) and median abso- 
lute relative error (MARE) for key parameters. The RE and 
MARE for each EM within a case were calculated as follows: 
RE i+ = (Ei it = ijt) / Das. and (1) 
MARE; , = median( | RE; «| ), (2) 
where E = the estimated quantity of interest; 
T = the true value from the OM; 
i = the quantity of interest; 
j = the iteration number; and 
t = the year, if applicable. 
For evaluating performance related to making stock sta- 
tus determinations (i.e., using F/F};,,;, and SSB/SSB;;,,i4), 
the accuracy of each model under a case equals the num- 
ber of correctly identified positives and negatives divided 
by the total number of iterations. 
Results 
Code comparison process 
Identification of common features The structure of the 
OM and cases were defined on the basis of the similar- 
ities and differences found among the 4 EMs (Table 2). 
On the basis of the comparisons, we found that the com- 
mon available spawner-recruit model among the EMs is 
the compensatory Beverton—Holt spawner-recruit model 
(Beverton and Holt, 1957) with lognormally distributed 
recruitment deviations. For selectivity patterns, all of the 
EMs have both a simple-logistic function and a double- 
logistic function. For the double-logistic function, the 
ASAP, BAM, and SS each require 4 parameters, and the 
AMAK requires 3 parameters. Bias adjustment of esti- 
mated mean recruitments is implemented in only the 
BAM and SS. For some features (e.g., available selectivity 
patterns), some of the EMs have more options than those 
listed in Table 2. We limited the options to features avail- 
able in at least 2 EMs. All available options can be found 
in the technical manual for each EM. Differences among 
other features have been summarized in Dichmont et al. 
(2016), Dichmont et al.', and Punt et al. (2020). 
Basic settings to ensure similar configurations Results of 
the comparison of formulas used in source code for key 
features indicate that analysts can manually adjust some 
basic settings to ensure that all 4 EMs are configured sim- 
ilarly and to ensure that a comparison study is effective. 
For example, in the AMAK, ASAP, and BAM, the popu- 
lation starts at age 1, but in SS the population routinely 
starts at age 0 and can be configured to start at older ages, 
as was done for this study. Survey index units, biomass or 
number of fish, can be used as input for the ASAP, BAM, 
and SS. In the AMAK, the default unit is biomass, but 
numbers could be used by setting all entries in the weight- 
at-age matrix to the value of 1. 
Selectivity-at-age outputs can be used directly for compar- 
ison, but estimated selectivity parameters need to be further 
converted before being compared because they are modeled 
differently in EMs. The simple logistic selectivity in the AMAK 
and BAM share the same formula as the OM (..e., equation 
5.1 from Supplementary Table 1 [online only]). In the ASAP 
and SS, simple-logistic selectivity is calculated as follows: 
2 1+6e 2 wie 
1 : 
Sp = ja OOS respectively, (4) 
where S, = fishery selectivity at age; 
Ly = age at 50% selection parameter (equal to x, 
from equation 5.1 in Supplementary Table 1 
[online only], where x. is age at 50% selection 
parameter); 
Uy = Slope parameter (equal to 1/x, from equation 
5.1 in Supplementary Table 1 [online only]); 
a = ages; 
Vv, = age at 50% selection for fishery parameter 
(equal to x, from equation 5.1 in Supplemen- 
tary Table 1 [online only]); and 
Vv, = slope parameter (equal to In(19)/x, from equa- 
tion 5.1 in Supplementary Table 1 [online only]). 
Similarly, the BAM and SS share the same double-logistic 
selectivity formula (Equation 5), and the formula from the 
AMAK (Equation 6) and ASAP (Equation 7) are different. 
Consequently, parameter values cannot be directly compared 
between models. However, the resultant curves and selectiv- 
ity at age can be compared with the following equations: 
1 
Sr See ie —_ ——_——_—_| and (5) 
a 1 + e182) 1 +e P14 Be) 
Sp = Sp / max(Sp ); 
¥1 =P1+ Po, (6) 
Yq =P, +41 + p3, and 
1 1 2. 
Sp, oy —In(19)(a — y;) I -In(19)(a — y2) 10.95"; and 
l+e © l+e 
Sp = aed Lahey | ee and (7) 
a a-py a—[3 
1+e 2 1lt@ 
Sp = Sp, / max(Sp ), 
where S; = fishery selectivity at age before rescaling to 
ensure that it peaks at 1; 
