Comparison of automated crater catalogs for Mars from Benedix et al. (2020) and Lee and Hogan (2021)

Crater mapping using neural networks and other automated methods has increased recently with automated Crater Detection Algorithms (CDAs) applied to planetary bodies throughout the solar system. A recent publication by Benedix et al. (2020) showed high performance at small scales compared to similar automated CDAs but with a net positive diameter bias in many crater candidates. I compare the publicly available catalogs from Benedix et al. (2020) and Lee & Hogan (2021) and show that the reported performance is sensitive to the metrics used to test the catalogs. I show how the more permissive comparison methods indicate a higher CDA performance by allowing worse candidate craters to match ground-truth craters. I show that the Benedix et al. (2020) catalog has a substantial performance loss with increasing latitude and identify an image projection issue that might cause this loss. Finally, I suggest future applications of neural networks in generating large scientific datasets be validated using secondary networks with independent data sources or training methods.