New York University computer scientists and students have developed a pair of web-hosted AI systems designed to increase the efficiency and accuracy of the research process—by spotting “hallucinations” in academic citations and by identifying appropriate experimental protocols for scientific studies.
“We think these two tools will be most useful to natural scientists and, eventually, social scientists who wish to design new experiments based on the most relevant existing protocols and want to be sure that papers they write and depend on fairly represent the literature they cite,” said Dennis Shasha, a professor of computer science at NYU’s Courant Institute School of Mathematics, Computing, and Data Science, who led both projects.
The tools are described in a pair of articles in the journal MDPI Information .
Hallucination Nerd permits users to check whether the claims a document makes about its citations are, in fact, backed up by the cited documents. It grades such claim-citation pairs as “Supported,” “Partially Supported,” “Not Supported,” “Contradicted,” “Cited article does not exist,” or “Could not access.”
“Checking all the claims an article makes about other articles helps both readers and authors to determine whether the citing article fairly represents the state of the art,” explains Shasha. “Accurately and efficiently evaluating previous research can only improve subsequent, related studies.”
Protocol Nerd aims to help bench scientists seeking experimental protocols for their research projects—for instance, a biologist looking for ways to make rice more drought tolerant. In this scenario, if a user asks for such a protocol in English (e.g. “Find protocols that test for drought tolerance in rice”), ProtocolNerd would first ask clarifying questions (e.g. “What is the growth stage of the rice plant?”) and would explain why it’s asking. Once the scientist answers these questions, ProtocolNerd would provide multiple protocols and give reasons why each protocol might be helpful—or, alternatively, might not be.
“Many hundreds of thousands of experimental protocols exist—but sorting through them is labor-intensive and error-prone,” observes Shasha, who adds that ProtocolNerd is currently aimed at molecular biology and chemistry but can be extended by a domain expert to other disciplines, such as psychology and economics, even if that domain expert is not a programmer.
“If an experimentalist wishes to answer a new scientific question and needs a protocol to address that question, finding the right protocol quickly is of great use,” he notes.
The researchers validated their models by drawing upon research databases.
For ProtocolNerd , the validation used scientific articles from PubMed and Europe PMC . For each article, the benchmark tested whether ProtocolNerd and other state-of-the-art systems could find the protocol used by the article based solely on the title and abstract of the article.
For HallucinationNerd , the validation made use of arXiv , a full text/open access repository of research papers. It gauged the tool’s ability to separate hallucinations from supported claims by testing its outputs against the contents of papers in the repository and compared its performance against existing systems.
“We designed testing frameworks that closely resemble randomized control trials for the same reason that bench scientists consider such trials the gold standard in their field: to eliminate bias and be able to test for statistical significance,” Shasha concludes.
The work was supported by NYU Wireless , an academic research center at NYU’s Tandon School of Engineering that backs advances in wireless communications, sensing, networking, and devices, and the US Department of Energy’s Advanced Research Projects Agency-Energy (DE-AR0002025 M0002).
# # #
Editor’s Note:
In November 2025, NYU announced the establishment of the Courant Institute School of Mathematics, Computing, and Data Science . The newly established school recognizes the storied history of the Courant Institute of Mathematical Sciences—and its strengths in both applied and pure mathematics—while encompassing NYU’s Center for Data Science and linking the computer science departments at Courant and the Tandon School of Engineering.
Information
HallucinationNerd: A Framework and Tool to Detect Hallucinations and Unsupported Claims
22-Oct-2026