A Harvard study found that Claude beat a carefully selected pool of human reviewers in side-by-side academic peer review tests, with the authors concluding that AI review now exceeds even unusually strong human peer review. The paper was written by Gary King, a university professor at Harvard and director of the Institute for Quantitative Social Science, and Steven Worthington, the institute’s director of data science.
The experiment used the same set of 124 papers and asked both humans and Claude to write review reports. According to the study, human reviewers did not win a single one of the seven review dimensions examined.
Harvard matched paid expert reviewers against Claude
The human side came from rapidPeer, a pre-submission review service created by Harvard’s Institute for Quantitative Social Science and later opened to several other institutions. The study describes these reviewers as outside domain experts who were paid per review and had incentives to do strong work because better performance could lead to more assignments.
They were required to read the full paper and produce reviews of 1.5 to 2 pages focused on helping authors improve the manuscript, not deciding whether it should be published. In a survey of 74 authors, 88% rated the review quality as high or very high. Several former editors from well-known journals also said the reports were among the best they had seen.
The authors wrote that, to their knowledge, no group of human reviewers had been asked to work under better conditions than these. The dataset covered all valid submissions to the service from 2019 to 2026: 124 papers and 134 human reviews spanning statistical analysis, mathematical modeling and purely qualitative research.
Claude entered in two modes, with GPT-5.5 as judge
Claude Opus 4.8 was used with maximum reasoning settings. It competed in two forms.
- One version worked under the same 13 review guidelines given to humans and was not allowed to use outside tools.
- The other was a tuned version with a dedicated review prompt and web access, averaging 12.4 searches per paper.
The prompt instructed Claude to read the full manuscript and figures closely, search for prior work the authors may have missed, verify every cited source with a real search and link, and avoid citing from memory. It also told the model to separate central problems from minor ones, focus first on the two or three most important issues, skip pleasantries and avoid padding.
The judge was GPT-5.5 from another company. Each pair of reviews had formatting and source information removed, then the order was swapped and compared twice. A win counted only if the same review was chosen both times. The judge also scored each review from 0 to 10 across seven dimensions.
Claude led on overall quality and every measured dimension
The authors described the outcome as unambiguous. In overall comparisons, GPT-5.5 did not award a single win to the human side. Every comparison that produced a winner went to Claude.
Across the seven dimensions, Claude swept three categories outright: identifying weaknesses, evaluating methods and adding literature. It won 99% of comparisons on figures and charts. It also won 92% in each of theory and ideas, argumentative structure, and expression.
Average scores showed the same gap. Claude received 8.7, while human reviewers averaged 6.0.
The widest differences appeared in two of the most labor-intensive parts of reviewing:
- Literature coverage: human reviewers averaged 4.73, while web-enabled Claude scored 9.55. Without web access, Claude won only 62% of comparisons in this category. With search turned on, it swept the field.
- Figures and charts: humans averaged 2.95, versus 7.57 for Claude. The study said many human reviews did not mention figures at all.
The paper also highlighted an alignment result. The three former editors agreed with one another 80% of the time, but their agreement rate with GPT-5.5 was 87%. In other words, the editors matched the AI judge more often than they matched their own peers.
Former journal editors picked Claude 26 times in 30 blind choices
The study did not rely only on model-based judging. It also asked three former editors from well-known journals to make blind selections. Each editor reviewed 10 pairs of anonymous review reports and saw only the reports, not the original papers.
Out of 30 total choices, the editors selected Claude 26 times. One of the three editors did not choose the human review in any of the 10 pairs that person evaluated.
Ten robustness checks did not reverse the result
The authors anticipated objections and ran ten classes of robustness checks.
One concern was that AI might simply be winning by writing longer reviews. The researchers restricted the sample to 44 pairs with similar length. Claude still won all 44.
Another concern was that GPT-5.5 might prefer an AI writing style. To test that, the researchers rewrote all 134 human reviews in an AI-style voice while keeping the arguments unchanged. Those rewritten reviews still failed to win a single comparison, and their average score slipped from 5.93 to 5.87.
A third concern was that the human sample might include mediocre work. The researchers then kept only the top 40% of human reviews by score and selected the 28 pairs closest to AI performance. Humans still did not win a single pair, and the best human review still scored 0.7 below the AI review.
Another question was whether Claude had seen later published versions of the papers during training. When the sample was split by submission year, tool-free Claude scored 8.2 on papers from 2022 and earlier and 8.1 on papers from 2025 to 2026, a negligible difference.
The authors also tested whether the finding depended on one model and one judge. They had Claude and GPT alternate as reviewer and judge in 416 comparisons. No matter which model judged, AI won more than 90% of the time, and the human review was usually near the bottom in a five-way ranking.
Claim-by-claim checks against the manuscripts found humans wrong far more often
The study then asked a harder question: not which review sounded better, but which one was more accurate.
First came citation checks. Web-enabled Claude cited 651 references with DOI or arXiv identifiers in its reviews. The researchers checked them one by one against databases and said none of the 651 references was fabricated. In a spot check of citation usage, 99 out of 100 assessable cases matched the issue being discussed.
The more demanding test examined statements about the manuscripts themselves. This round compared tool-free Claude with human reviewers. The researchers broke reviews into sentence-level claims, removed source information and had a model judge each claim against the original paper. The threshold for calling something wrong was strict: there had to be direct contradiction in the manuscript.
The paper gives an example of a reviewer criticizing a manuscript for lacking robustness checks when Section 5 explicitly included them.
On that basis, human reviewers were judged wrong in 152 of 956 checked claims, or 15.9%. Claude was judged wrong in 57 of 1,039 checked claims, or 5.5%. Measured at the review level, 64.2% of human reviews contained at least one error, compared with 37.1% for Claude.
The researchers also tested the reliability of that checking system. In a manual review of 90 judgments, 87 matched the model’s conclusion. The remaining three disagreements were all cases where the model had wrongly accused Claude. They also inserted 258 trap sentences that reversed the meaning of review statements by changing only a few words. All of those false statements were caught, while the same number of true statements produced no false positives.
For the authors, that matters because these human experts were paid, reputation-sensitive and required to read the full paper, yet they still produced a much higher error rate when their comments were checked against the source text.
The paper sketches a workflow where AI reviews come before submission
The second half of the paper turns to journal policy. It argues that reviewers are already using AI whether journals allow it or not. The study cites Nature as reporting that more than half of researchers use AI in peer review, with many doing so against existing rules. It also cites a randomized experiment at ICML 2026 in which 22.5% of respondents assigned to a “no AI” group still admitted using it.
Stanford political science associate professor Xu Yiqing, sharing the paper, said journals that ban AI in peer review should rethink that position. Wharton professor Ethan Mollick also reposted the study and urged academia to work out how scientific research should adapt to AI.
King and Worthington propose a possible workflow. Journals would publish an official AI review prompt, and authors would use it to review and revise their papers several times before submission. At submission, authors would provide the manuscript, the AI review and a point-by-point response. Editors would then run a second review using a different, unpublished prompt. Hard cases would go to editorial board members or outside experts. The final publication decision would still rest with editors.
The authors say such a process could cut publication decisions from months to days.
They also argue that author response letters would remain essential because AI can become excessively literal late in the revision cycle. The paper gives an example: AI might say, “Yes, 2+2=4, but you never defined the + sign.” In that situation, authors need a formal place to push back.
The paper does not ignore risks. The authors write that individuals using AI to generate ideas may each look more original on their own, yet the combined output may grow more similar over time and drift toward mainstream, conservative positions. Their recommendation is that journals should keep tracking cases where authors and AI disagree, then keep adjusting prompts and models.
Editors, not models, still make the publication call
The paper ends on a narrower claim than “AI should replace journals.” Serious, detailed and useful review reports have long been scarce in research, often taking months to arrive and depending heavily on reviewer effort. If AI can make that kind of feedback available on demand, researchers could hear criticism earlier and revise faster.
But the authors still draw a line at editorial judgment. Deciding which paper belongs in which journal, they argue, remains a task for human editors.
The references listed in the source material include the paper page on Gary King’s website and Ethan Mollick’s post on X. The MarsBit article says the piece originally came from the WeChat public account “新智元,” written by ASI启示录 and republished by MarsBit.