Two alternative evaluation metrics to replace the true skill statistic in the assessment of species distribution models

By

Wunderlich, Rainer Ferdinand
Lin, Yu-Pin
Anthony, Johnathen.
Petway, Joy R.

Type

Article

Date of Publication

2019

Original Publication

Nature conservation

Volume

35

Pages

97-116

Subjects

Evaluation , presence-background , species distribution modelling , True Skill Statistic

Contributed by

Pensoft Publishers

Language

English

Published by

Sofia, Bulgaria Pensoft Publishers 2012-

Identifiers

DOI: https://doi.org/10.3897/natureconservation.35.33918

 

Abstract

Model evaluation metrics play a critical role in the selection of adequate species distribution models for conservation and for any application of species distribution modelling (SDM) in general. The responses of these metrics to modelling conditions, however, are rarely taken into account. This leads to inadequate model selection, downstream analyses and uniformed decisions. To aid modellers in critically assessing modelling conditions when choosing and interpreting model evaluation metrics, we analysed the responses of the True Skill Statistic (TSS) under a variety of presence-background modelling conditions using purely theoretical scenarios. We then compared these responses with those of two evaluation metrics commonly applied in the field of meteorology which have potential for use in SDM: the Odds Ratio Skill Score (ORSS) and the Symmetric Extremal Dependence Index (SEDI). We demonstrate that (1) large cell number totals in the confusion matrix, which is strongly biased towards ‘true’ absences in presence-background SDM and (2) low prevalence both compromise model evaluation with TSS. This is since (1) TSS fails to differentiate useful from random models at extreme prevalence levels if the confusion matrix cell number total exceeds ~30,000 cells and (2) TSS converges to hit rate (sensitivity) when prevalence is lower than ~2.5%. We conclude that SEDI is optimal for most presence-background SDM initiatives. Further, ORSS may provide a better alternative if absence data are available or if equal error weighting is strictly required.