AI tools flag reproducibility and error issues across top research papers

AI-based review systems are beginning to expose how often published research may contain reproducibility problems or objective mistakes, according to a MarsBit report citing several recent projects focused on top artificial intelligence venues. One audit, carried out on all 168 oral papers at ICML 2026 by a U.S. research review company, found that 92 papers had at least five verifiable claims. Among those, 58 could not be reproduced. Only 34 papers had more than 40% of their claims successfully reproduced, and just eight cleared the 80% mark. The report draws a line between failed reproduction and fraud. In the cases cited, failures were tied to missing files, broken software dependencies, mismatches between paper claims and runtime results, and four papers that depended on models no longer available. A separate GPT-5-based paper checking system, introduced in late 2025 to scan already published papers in top AI conferences and journals, reported an average of 4.7 objective errors per paper, with 99.2% of papers flagged for at least one issue. The report says mathematics and formula-related mistakes made up 54.0% of all errors, while 30.8% of NeurIPS papers and 23.8% of ICLR papers contained at least one substantive error that could affect interpretation. Even so, the article notes that AI review tools still need human oversight: one checker cited had 83.2% precision and still missed about 40% of real errors in each run.