A Southwest Research Institute-led investigation has uncovered significant discrepancies in artificial intelligence-generated lunar crater catalogs, warning planetary scientists against relying on published performance metrics without rigorous independent validation. The study, conducted by Dr. Stuart J. Robbins and co-authored by Dr. Rachael H. Hoover at the institute’s Boulder, Colorado facility, systematically evaluated eight AI-derived crater databases against a comprehensive, manually compiled lunar catalog. The results demonstrated that standardized automated metrics frequently overstate the accuracy of machine-learning tools, with performance scores dropping dramatically when measured against strict geological criteria. Impact crater catalogs serve as foundational records for reconstructing the geological timelines of planetary bodies. By analyzing crater density, spatial distribution, and dimensions, researchers estimate surface ages across the solar system. Automated detection promises to accelerate this traditionally labor-intensive process. However, the SwRI analysis reveals that current AI models often fall short on scientific precision. Researchers found that algorithmic outputs frequently exhibit positional shifts, size inaccuracies, and duplicate entries. Such errors carry direct scientific consequences; for example, artificially duplicated craters can double the projected age of a planetary surface, fundamentally distorting geological models. The evaluation further exposed a critical flaw in relying on single summary metrics. AI databases may appear highly effective in aggregate, but their accuracy varies significantly depending on crater diameter. Performance degrades noticeably for smaller impact structures, limiting their utility for specific analytical tasks. The team emphasized that computer vision benchmarks often mask these deficiencies by prioritizing broad detection rates over the spatial and dimensional precision required for rigorous planetary science. Despite these limitations, the researchers stress that the findings do not undermine the value of artificial intelligence in space exploration. Instead, they outline a necessary pathway for reliable integration. The authors advocate for the establishment of uniform scientific benchmarks, transparent reporting of matching criteria, and independent validation protocols. They recommend that laboratories and mission planners treat AI-generated catalogs as preliminary tools rather than finalized datasets until they consistently meet human-expert standards. As planetary exploration increasingly depends on large-scale geological analysis, standardizing how AI performance is measured will determine whether machine learning fulfills its promise of accelerating scientific discovery or introduces systemic errors into celestial mapping. The SwRI team concludes that while automation can drastically reduce manual workload and unlock previously unaddressable research questions, its deployment must be guided by disciplined verification and rigorous scientific scrutiny.