SAN ANTONIO — July 6, 2026 — A new Southwest Research Institute-led study compared eight AI-generated lunar crater catalogs, discovering that many of their published performance metrics drop sharply when the databases are evaluated using the same scientific standards humans are held to. Crater catalogs provide a comprehensive record of impact craters on planetary surfaces. They log the precise location, dimensions and physical characteristics of impact structures to help scientists understand the geological history of the solar system and its components.
Impact craters are the dominant geologic feature on the Moon and many other solid worlds. Scientists estimate the ages of planetary surfaces by counting impact craters. Because small asteroids strike at a roughly steady rate, surfaces with more craters are older than those with fewer. By measuring crater sizes and spatial densities, and knowing how often craters form, researchers can estimate surface ages on worlds throughout the solar system. Accordingly, scientists use crater catalogs to reconstruct geologic histories and study how planetary surfaces evolve. Automated crater detection — using artificial intelligence and machine-learning — could save researchers years of painstaking manual work and help tackle problems otherwise impossible to address in a lifetime.
“AI has enormous potential to help with repetitive, time-consuming scientific tasks, especially gathering some of our data,” said Dr. Stuart J. Robbins of SwRI’s Solar System Science and Exploration Division in Boulder, Colorado, and lead author of the study. “But our analysis shows that researchers should not assume an AI-generated crater catalog is ready for scientific use solely based on its published metrics.”
The study, “A Comparison of Lunar AI-Based Crater Databases Using Uniform Criteria,” compared eight global or large-coverage lunar crater catalogs generated using automated methods. Robbins and co-author Dr. Rachael H. Hoover, also of SwRI, evaluated each database against a large, manually compiled lunar crater catalog that took Robbins years to construct, applying the same matching criteria to each AI catalog.
The team found that performance depends strongly on how a crater “match” is defined. Candidate craters must be in the right place and be sized accurately to be useful for many planetary science applications. Some common computer-vision metrics can make automated detection look acceptable even when a crater’s size or location is scientifically inaccurate.
“A crater catalog is not just a random list of circles,” Robbins said. “If a crater is shifted, duplicated or improperly sized, that can affect the science that depends on those metrics. For instance, if a surface with a model age of 1 million years requires x number of craters and AI accidentally duplicates those craters, suddenly the model would double the surface’s projected age.”
The team applied stricter criteria based on the repeatability of manual crater analysts, nearly all databases with published metrics performed worse than reported, with some values dropping by more than a factor of 10. The study also discovered that single summary metrics can hide flaws in the data. Some databases performed relatively well for certain crater sizes but poorly for others.
“Diameter dependence matters,” Robbins said. “A catalog might look acceptable from one overall number, but when you break it down by crater size, it may be useful for one question while unreliable for many others.”
The researchers emphasized that the study is not an argument against using AI in planetary science. “Our work highlights the necessary next step of standardizing benchmarks, including transparent reporting of matching criteria and independent validation, so AI-generated catalogs can be properly used for scientific analysis,” Hoover emphasized.
“AI may eventually transform crater cataloging and revolutionize how we gather our science data — potentially saving years of time,” Robbins added. “For now, researchers need to not chase it as the solution to everything. We need to understand how these tools work, where they fall short and whether their performance is good enough to support the science being done.”
To see an animation comparing human- and AI-generated crater catalogs, see: https://youtu.be/Z5BiesDc7eo .
For more information, visit https://www.swri.org/markets/earth-space/space-research-technology/space-science/planetary-science .
The Planetary Science Journal
Data/statistical analysis
Not applicable
A Comparison of Lunar AI-Based Crater Databases Using Uniform Criteria
6-Jul-2026