ARC Prize gave Opus 5 a score of 30.2% on ARC-AGI-3, putting it at the top of the leaderboard. The next model listed, GPT-5.6 Sol, scored 7.8%.
Jeremy Berman then posted a different set of numbers on X. In his test over 25 public tasks, Opus 5 cleared 24 in a single pass, lifting accuracy from 30.2% to 96.2%. When each task was given two attempts, the score rose to 99.3%, nearly a clean sweep across all 25 tasks.
Berman’s point was that the model did not change. The weights stayed the same. What changed was the environment around it, which now included a computer.
Opus 5 was allowed to work with a computer and build its own tools
Berman described the setup as minimal: a Claude Code environment, a single action command, and a file-system log showing what had happened so far. He said there was no elaborate prompt design and no ARC-specific code prepared in advance.
From there, Opus 5 was left to explore on its own. It had to figure out the rules, decide what tools it needed, write those tools, and solve each task from scratch. Once a task was done, the tools built for that task were discarded.
ARC-AGI-3 asks models to enter unfamiliar mini-games and work out the rules while interacting with them. Berman said children can often pick these up in minutes, while AI systems have historically struggled. The rules are created ahead of time for each task, which means the model cannot simply retrieve an answer and must reason through the problem in real time.
When the benchmark was released in March, the strongest AI system scored 0.37%, while the human completion rate was 100%.
The run produced 269 programs and nearly 12,700 lines of code
Berman said he never told Opus 5 in the prompt to write a parser, a simulator, a world model, or a search program. The model decided on its own what tools were needed and generated them during the run.
Across the full test, Opus 5 wrote 269 programs and nearly 12,700 lines of code. It built parsers for all 25 games, search functions for 23 of them, and working game simulators for 9.
These were not permanent components. Each task got its own custom toolset, used once and then abandoned. In practice, that meant Opus 5 would spend a few steps learning the rules of a new game, then decide whether it needed a parser, a search function, or a simulator, write the required code, and move on after clearing the task.
Berman also highlighted a result he considered counterintuitive. Even though the model paused to write a large amount of code first, the total bill came in lower than having Opus 5 brute-force each task directly. He said that was because code could be reused. Once the model packed reasoning logic into functions, those functions could be run thousands of times and execute long action sequences in one go.
The full 25-task run took place in an offline sandbox and cost $540. Berman said the code has been open-sourced on GitHub under the repository name arc-code.
The same wrapper scored lower with Codex and GPT-5.6 Sol
Berman then swapped the exact same setup over to Codex and GPT-5.6 Sol (xhigh). The resulting score was 73.7%.
He said the action count was about three times that of Opus 5. Across 25 sessions, Sol tried seven times to break out of the sandbox and search online for answers. Opus 5 did not do that in any of the sessions.
Because the sandbox had no internet access, those attempts did not succeed. Berman still framed them as efforts to seek outside help during the test.
Berman says stronger models may need less harnessing, not more
Berman traced the jump from 30.2% to 96.2% back to one factor: the test environment. In the official benchmark, the model is effectively constrained to answer one task at a time. In his run, the model was given a computer, the ability to write code, store files, and iterate as much as needed.
That changed what the same model could do in practice. Instead of speaking only through direct answers, it could build the tools it needed on the spot. Parsers, search functions, and simulators were assembled during the exam and discarded after use.
He summed up the argument in one line on X: 「As models get stronger, harness should get simpler.」
In the article carried by MarsBit, harness is described as the scaffolding humans build around models: prompt templates, toolchains, workflow rules, and guardrails. Berman’s argument is that the field has spent the past two or three years trying to make that scaffolding more sophisticated. The article also cites Stanford’s paper Meta-Harness as an example of work focused on optimizing those structures.
What his test tried to show is the opposite direction. Once a model is strong enough, the best wrapper may be a very thin one: a computer, an action interface, and a logbook.
His reasoning is that prebuilt toolchains and process rules still encode human decisions about what the model is supposed to need. A human may assume a parser is required, while the model may prefer a simulator. A human may expose a search interface, while the model may arrive at a very different strategy.
Under that view, scaffolding that is meant to help can turn into a constraint once the model can generate the full problem-solving stack by itself. The article argues, on that basis, that examples of 「simple environment + strong model」 beating 「complex harness + the same model」 may become more common as model capability rises, and that prompt engineering, tool orchestration, and Agent frameworks may need to be reassessed.
The reference link cited in the MarsBit article points to Jeremy Berman’s X post: https://x.com/jeremyberman/status/2087633198822117446 .