USTC moved AI from planning into a physical lab
A new study from the University of Science and Technology of China asks a narrower question than the usual “can AI do science?” debate: is an AI system smart enough to carry out scientific research end to end and make autonomous discoveries?
Instead of limiting the test to question answering or experiment generation, the team connected the AI “brain” to the “body” of a machine-scientist lab and pushed the system into real-world experiment execution and feedback learning.
The paper is available at https://arxiv.org/abs/2607.23045.
The team built a machine-catalysis laboratory with 45 modular automated workstations spanning synthesis, characterization, and catalytic performance testing. To make the lab understandable to AI, the researchers packaged its capabilities as machine-readable skills. Agents could then call equipment directly, while still being bound by the hardware’s actual limits, operating rules, and experimental conditions.
Figure 1 shows the AI-readable machine-scientist lab architecture for catalysis research, while Figure 2 maps the path from scientific intent to machine execution.
48 configurations, 4,608 trials
Using this platform, the researchers ran a systematic evaluation of 48 real-world configurations built from six agent frameworks and nine large language models. The benchmark covered 32 research tasks defined by domain experts, adding up to 4,608 evaluation runs.
The review did not stop at whether an agent could write an experimental plan. It also tracked whether the plan could pass verification, be sent to the robotic system, and run end to end without human intervention.
Execution rates were still low
The results show that even leading LLM agents are still far from taking over a lab. Out of 4,608 tests, only 151 workflows could run without manual repair, equal to 3.3% of the total.
The strongest result came from Claude Code paired with Claude Opus 4.7, which reached a 28.1% execution rate. Codex paired with GPT 5.5 followed at 19.8%.
Figure 3 shows agent performance across executable planning, verification, and task dispatch to the lab.
Adjusting parameters is not the same as redesigning research
The study also tested whether agents could learn from live experimental results. The team placed the Codex/GPT 5.5 pair into an open five-round loop of planning, robotic execution, evidence collection, and replanning.
The agents could adjust material formulations and operating conditions after receiving results, but those changes stayed at the level of local parameter tuning. Across the five rounds, the workflow skeleton remained intact. The system did not redesign the analysis method, and it failed to correct persistent omissions such as a missing electrode binder and a color reagent tailored to a specific analyte.
That means reading experimental feedback and changing parameters is not the same as identifying a flaw in the research strategy, and it is not the same as scientific replanning.
Long-horizon planning is still the bottleneck
Some agents did produce expert-rated executable workflows with as many as 44 operational steps, but across the full benchmark only three workflows went beyond 30 steps. The longer the task chain, the harder it became for agents to preserve both logical consistency and physical executability.
The paper splits “AI scientist” into three abilities
The study separates three capabilities that are often blurred together in AI scientist discussions: generating an experimental plan fluently, producing a workflow that can actually run in a physical lab, and changing the overall research strategy in response to experimental results.
The results suggest that language-level planning does not automatically turn into reliable lab execution. Local parameter tuning also should not be treated as scientific replanning.
Machine labs can be both testbeds and training grounds
As an AI-for-science research infrastructure, a machine lab can serve as a testbed for scientific ability and as a training ground for future systems. Through machine-readable skills, agents can speak directly with the lab, turn scientific intent into executable tasks, and receive feedback from execution, instrument status, and experimental results.
The resulting planning-execution-feedback-replanning loop can expose gaps in knowledge, operations, and strategy. It can also turn successful workflows, failed cases, experimental results, and expert reviews into training and alignment data for future model and agent iteration.
Figure 4 shows five rounds of iteration for AI-machine scientists under an open scientific problem.
Reference: https://arxiv.org/abs/2607.23045
This article is based on the WeChat account “新智元”, authored by 新智元 and edited by LRST.