Geographic outliers at GBIF (Global Biodiversity Information Facility) are a known problem. Outliers can be errors, coordinates with high uncertainty, or simply occurrences from an undersampled region. Often in data cleaning pipelines, outliers are removed (even if they are legitimate points) because the researcher does not have time to verify each record one-by-one. Outlier points are usually occurrences that need attention. Currently, there is no outlier detection implemented at GBIF and it is up to the user to flag outliers themselves.<br> DBSCAN (a density-based algorithm for discovering clusters in large spatial databases with noise) is a simple and popular clustering algorithm. It uses two parameters, (1) distance and (2) a minimum number of points per cluster, to decide if something is an outlier. Since occurrence data can be very patchy, non-clustering distance-based methods will fail often Fig. 1. DBSCAN does not need to know the expected number of clusters in advance. DBSCAN does well using only distance and does not require some additional environmental variables like Bioclim. <br> Advanatages of DBSCAN :<br> <br> <br> <br> Simple<br> <br> <br> Easy to understand<br> <br> <br> Only two parameters to set <br> <br> <br> Scales well<br> <br> <br> No additional data sources needed<br> <br> <br> Users would understand how their data was changed<br> <br> <br> <br> Simple<br> Easy to understand<br> Only two parameters to set <br> Scales well<br> No additional data sources needed<br> Users would understand how their data was changed<br> Drawbacks : <br> <br> <br> <br> Only uses distance<br> <br> <br> Must choose parameter settings<br> <br> <br> Sensitive to sparse global sampling <br> <br> <br> Does not include any other relevant environmental information<br> <br> <br> Can only flag outliers outside of a point blob <br> <br> <br> <br> Only uses distance<br> Must choose parameter settings<br> Sensitive to sparse global sampling <br> Does not include any other relevant environmental information<br> Can only flag outliers outside of a point blob <br> Outlier detection and error detection are different. If your goal is to produce a system with no false positives, it will fail. While more complex environmentally-informed outlier detection methods (like reverse jackknifing (Chapman 2005)) might perform better for certain examples or even in genreal, DBSCAN performs adequately on almost everything despite being very simple. <br> Currently I am using DBSCAN to find errors and assess dataset quality. It is a Spark job written in Scala (github). It does not run on species with lots of (&gt;30K) unique latitude-longitude points, since the current implementation relies on an in-memory distance matrix. However, around 99% of species (plants, animals, fungi) on GBIF have fewer than &gt;30K unique lat-long points (2,283 species keys / 222,993 species keys). There are other implementations ( example) that might scale to many more points. <br> There are no immediate plans to include DBSCAN outliers as a data quality flag on GBIF, but it could be done somewhat easily, since this type of method does not rely on any external environmental data sources and already runs on the GBIF cluster.
