162 
-02 00 02 04 -0.4 -0.2 
RE in MSY 
RE in Fyysy 
—4- AMAK 
—- ASAP 
Fishery Bulletin 119(2-3) 
0.2 0.4 -0.2 00 02 04 
RE in SSBysy 
—<— BAM 
—— SS 
Figure 5 
Relative error (RE) in maximum sustainable yield (MSY), fishing mortality rate that corresponds to MSY 
(Fysy), and spawning stock biomass at MSY (SSBysy) for 4 age-structured estimation models under cases 
0-12 (CO-—C12). The models, evaluated in this study for use in stock assessments, include the Assessment 
Model for Alaska (AMAK), the Age Structured Assessment Program (ASAP), the Beaufort Assessment 
Model (BAM), and Stock Synthesis (SS). The bar is equivalent to the whisker of a box plot. The left bar rep- 
resents the smallest value larger than 1.5 times the interquartile range below the first quartile. The right 
bar represents the largest value smaller than 1.5 times the interquartile range above the third quartile. The 
symbols in the center represent median RE. 
receiver operating characteristic curves may also help 
summarize the overall degree of agreement (e.g., true pos- 
itive and false negative classifications, accuracy rate, error 
rate, and sensitivity) between estimates and true status 
(Connors and Cooper, 2014; Cortés and Brooks, 2018). 
Examination of the value of the code comparison process 
This study is unique among other model comparison stud- 
ies because a code comparison process was included before 
the simulation-estimation process. Identifying common fea- 
tures among the EMs and comparing source code can 
reduce contamination by parameter misspecification or 
analyst effect if the parameters are fixed in a comparison 
study. Although we estimated selectivity in this comparison 
study, during the early stage of the study, we fixed the selec- 
tivity at the true values. It is important to carefully apply 
fixed values because, in the 4 models, simple logistic selec- 
tivity was defined with parameters that had the same name 
(e.g., slope) but different interpretations. Otherwise, 
