An OpenAI researcher’s comment — “we don’t read papers anymore” — has stirred a wider argument over how much weight top AI conference papers still deserve.
The immediate trigger was an ICLR paper whose results looked unusually strong. After people online dug into it, they said they found a hidden “secret” behind those results, pushing a broader question back into view: how many papers from marquee conferences can actually hold up under scrutiny?
SAI reviewed all 168 ICML 2026 oral papers
In July, SAI, a group co-founded by University of Chicago computer science and data science associate professor Tan Chenhao, published what it described as a large-scale experimental review.
Its target set was every oral paper accepted at ICML 2026. The conference received 23,918 submissions this year, and only 168 were selected for oral presentation, about 0.7% of the total. In other words, the audit was focused on a slice that had already cleared one of the tightest filters in the field.
SAI Review did more than read the papers’ methods, experimental designs and reported results. It also downloaded code, models and data, set up the environment, ran the experiments, and compared the outputs line by line with what the papers claimed.
Out of the 168 oral papers, only 104 had open-sourced code, and SAI ultimately completed 105 full reproductions. The outcome was weak: only 34 papers reproduced more than 40% of their claims, and only eight reproduced more than 80%.
The report said median reproduction scores stayed in the 28% to 30% range whether a paper had at least one, three or five verifiable claims, with little change in the overall distribution.
Even after excluding experiments that were not run, were stopped early, or exceeded hardware limits, the median score was still only 42% to 50%. When each paper had at least three verifiable claims, the median held at 42%.
Recurring problems were basic and concrete
SAI said the most common failures were familiar ones: code that would not run, missing files, incomplete instructions, broken dependencies, or results from the code that did not match what the paper reported.
Four papers also depended on models that had already been taken offline. That means later researchers could not obtain the same results again even if they were willing to spend both time and money.
SAI highlighted two specific examples.
One paper promoted the claim that it trained only 0.77% of a base model’s parameters. The released checkpoint, however, had trained 6.31%, roughly eight times the stated figure.
Another paper presented a reliability table scored by a judge model, but the open-source code did not include that judge model, and there was no script capable of generating the numbers shown in the table.
High reproduction costs make outside verification harder
The article argued that the published reproduction results were already poor. The deeper problem, though, is that these are only the issues that can be detected under favorable conditions.
Papers without code are nearly impossible for outsiders to inspect. Even when code is public, checking a paper can demand large amounts of time, labor and money, with no guarantee of a clean answer at the end.
Using public on-demand pricing from Google Cloud, SAI estimated that the median cost of fully rerunning a single ICML oral paper was about $8,900. Among the 105 papers, 17 would cost more than $100,000 to reproduce, and the most expensive one came close to $2.2 million.
The more a paper depends on large-scale compute, the harder independent reproduction becomes. Without code, others cannot check it. With code, they still may not be able to afford the attempt.
Under those conditions, a flawed paper can still pass peer review, gather citations, go onto a CV, and help someone win admission, a faculty post, or a role at a major lab. Even if later work finds that the results do not line up, conferences rarely reopen review, and papers are not always retracted.
Papers are questioned as knowledge, but still used as a gate
In the large-model field, some important advances no longer appear first in papers. From the vantage point of an OpenAI engineer, that is not hard to understand: frontier labs have more compute, faster experimental loops, and a large body of internal results that never become public.
Still, the statement landed awkwardly because the career system around papers has not gone away.
One comment mocked this attitude as pulling up the ladder after getting in: companies recruit researchers from universities, build on years of openly accumulated academic work, use higher pay to compete for talent and GPUs, submit less of their own work to peer review, and then turn around and say academic research is mostly a scam.
The line is harsh, but the article said it contains a real point. People inside frontier labs may feel free not to read papers, yet people trying to enter those labs still need papers first.
For students and young researchers without industry experience, top-conference papers remain the most direct proof of research ability. They still matter for PhD applications, faculty hiring, and gaining attention from labs such as OpenAI.
Industry researchers may downplay papers once they are inside major labs, but they still use paper counts, conference prestige and citation records to screen the people outside the door.
That leaves papers in an awkward position: their credibility as vehicles for knowledge is under pressure, while their value in talent competition remains intact.
A tension between incentives and ideals
The article ended by arguing that the claim that “big labs don’t read papers anymore” can sound aloof because the people who can afford to stop relying on papers often crossed that threshold through papers in the first place.
At the same time, it did not portray every student as a deliberate fraudster. A university researcher quoted in the piece joked that he wished his students were already capable of writing a paper that exaggerated claims or even faked them.
Behind that joke sits a blunt contrast: some people are already competing to become academic tricksters, while others are still struggling just to get through LaTeX.
Reference links
- https://x.com/MathewShen42/status/2084465434506768867
- https://x.com/kellerjordan0/status/2084721463089902074
- https://sai.science/blog/how-much-science-is-verifiable
- https://x.com/mengyer/status/2085134204786921886
The original article was published by the WeChat public account “机器之心” (ID: almosthuman2014) and credited to 机器之心.