AI is cutting the cost of checking published research, and that shift is bringing more old results back under scrutiny.
Several recent projects focused on top AI conferences suggest that reproducibility gaps and objective errors in published papers may be more widespread than commonly assumed.
AI Agent audit reviewed ICML 2026 oral papers
According to the report, a U.S. research review company used AI agents on July 22 to run a systematic reproduction audit of all 168 oral papers accepted at ICML 2026.
Of those 168 papers, 92 contained at least five verifiable conclusory claims. The results were weak. Only 34 papers had more than 40% of their claims successfully reproduced, and only eight had more than 80% reproduced. That means 58 of the 92 papers could not be reproduced.
The report stresses that failed reproduction is not the same as research fraud. It lists several reasons for failure, including missing key files in released code, broken dependency versions, runtime results that did not match what the paper reported, and four papers that relied on models that had already been taken offline, making those experiments impossible for anyone to reproduce.
Some examples were more direct. One paper promoted the fact that it trained only 0.77% of a base model’s parameters, but the released checkpoint showed that 6.31% had actually been trained, a gap of about eight times. Another paper included a reliability table built on a certain judge model, but the open-source code did not include that model and provided no script capable of generating the table’s results.
Separately, Hugging Face and AlphaXiv launched the Agent Reproduction Challenge for ICML 2026, inviting researchers to use AI coding agents to automate reproduction of accepted papers. The report says the outcome there was also poor.
GPT-5 checking system reported objective errors at scale
The article also cites a study released in late 2025 that developed a GPT-5-based paper checking system designed to analyze papers already published in top AI conferences and journals and look for objectively verifiable problems.
The researchers stated their scope clearly: “We only look at ‘objective errors.’ We do not judge novelty or research value.”
Its results showed an average of 4.7 objective errors per paper, while 99.2% of papers were flagged for at least one issue.
By category, mathematics and formula-related errors accounted for the largest share at 54.0%, including incorrect equations, flaws in derivation logic, and false assumptions in proofs. About 30.8% of NeurIPS papers and 23.8% of ICLR papers contained at least one substantive error that could affect interpretation of the results.
The time trend moved in the same direction. The average number of errors per NeurIPS paper rose from 3.8 in 2021 to 5.9 in 2025, an increase of 55.3%.
AI is lowering the cost of reviewing older literature
The report argues that checking prior literature, identifying mistakes, and publishing corrections have long been accepted forms of academic output. What has changed is cost. AI can now help researchers search, compare, and trace inconsistencies much faster than before.
It outlines several approaches. One is to shift attention from finding entirely new topics to finding anomalies in older papers, especially highly cited classics that have drawn relatively little controversy. Another is to use AI to build knowledge maps and research lineages that make it easier to see citation links, unresolved disputes, and widely cited claims that have not been thoroughly validated. A third is to revisit long-accepted assumptions, such as whether a classic experiment would still satisfy current standards of verification, or whether a widely cited conclusion contains overlooked limiting conditions.
The article’s point is simple: this kind of questioning used to be expensive because it required extensive manual review of original literature and painstaking comparison of experimental details. AI is reducing that burden.
From recent AI papers to century-old scientific data
The shift is not limited to AI research. The report cites Pios, a theoretical chemist at Zhejiang Lab, who found a clear conflict between AI-predicted molecular boiling points and a chemical database compiled 75 years ago.
Pios first checked his own model. After manually tracing the original literature, he concluded that the AI result was the correct one. He then used AI for more checks and found that another boiling-point measurement, roughly a century old and widely treated as authoritative in the field, was also wrong.
Those data points had already been cited repeatedly and absorbed into later research. The article links the long survival of such errors to the scale of modern scientific literature.
In AI alone, annual submissions to ICLR climbed from 1,013 in 2018 to 19,619 in 2026. At that scale, an early error can be repeated across later papers and move through citation chains until it starts to look like settled consensus.
Older publication dates, complex citation networks, the time required for review, and limited academic payoff have all made systematic rechecking rare. The report says that, at least technically, large-scale re-examination of legacy scientific literature is now becoming possible.
Human review is still required
The article also sets limits on what current tools can do. It cites Paper Correctness Checker, powered by GPT-5, with a precision rate of 83.2%. Even so, around 40% of real errors still went undetected in each run.
That leaves AI fact-checking tools short of serving as final judges of scientific literature. They can improve screening efficiency, but their outputs still need to be reviewed by humans.
In that sense, publication no longer marks the end of scrutiny. For some papers, it may only be the start of another round of verification.
The piece was originally published by the WeChat account Quantum Bit (ID: QbitAI) and attributed to author Cheng Qian.