Large models remain weak at atom-level manipulation even when they can parse materials knowledge, according to AtomWorld, a new benchmark released by researchers from the Suzhou Institute for Advanced Research of the University of Science and Technology of China, the University of New South Wales and other institutions. Presented at ICML 2026, the benchmark offers a different read on one of the field’s most accepted assumptions: that bigger models and more data will keep producing broader capabilities through Scaling Law.
The paper argues that this pattern does not carry over cleanly into atom-operation tasks constrained by physical rules. In short, understanding is not the same as doing. Models may read papers, infer materials properties and analyze crystal structures, but that does not mean they can reliably modify an atomic structure under explicit instructions.
The paper is available at https://arxiv.org/abs/2510.04704. The project homepage is https://masterai-eam.github.io/atomworld/, and the code repository is https://github.com/MasterAI-EAM/atomworld.
Benchmark targets atomic operations rather than theory questions
In materials research, daily work often involves highly concrete instructions: build a specific material’s (001) surface to simulate real-world boundaries, replace atoms at designated lattice sites for doping or modification, or insert new atoms into selected interstitial positions to design storage and transport channels. Those tasks require a different capability set from text QA. They depend on 3D spatial control that remains consistent with physical laws.
AtomWorld was built to measure that ability directly. The framework uses standard crystallographic information from materials science to automate evaluation. It does not test recognition tasks or theoretical reasoning. Its focus is narrower and more practical: can a model change atomic arrangements precisely as instructed?
According to the study, the AtomWorld pipeline starts with a random sampler that retrieves preset atomic structures. A random initializer then assigns atomic indices and position parameters. Structural operators generate target structures, and a prompt module turns those into natural-language instructions. The resulting structure-text pairs are sent to large-model agents, and pymatgen’s StructureMatcher compares the model output with the target structure to score performance.
The researchers present this as a way to break down the broad complaint that “materials agents cannot model structures” into measurable atomic skills, from element replacement to spatial region judgment and continuous geometric understanding.
Scaling helps on templated tasks, then starts to fade
AtomWorld tested Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro, Gemini 2.5 Pro, Qwen3-32B, GPT o3, GPT-4o-mini, DeepSeek Chat and Llama3-70B, while also comparing Qwen models of different sizes.
The overall pattern was mixed. Larger models did better on some operations with clearer rules and more fixed solution paths, including atom replacement, deletion and movement. In the Qwen family, success rates on those tasks improved visibly from 4B to 32B, suggesting that scale still brings useful gains in settings where the rules are explicit.
But the same improvement did not generalize across the benchmark. When tasks depended on 3D spatial relations and geometric planning, larger size no longer produced stable gains. The paper highlights “rotation around an atom” as the clearest case. Performance stayed low across Qwen models of different sizes, and even Claude Opus 4.6, one of the stronger general-purpose models in the test set, posted only about a 12% success rate on that task.
The same kind of bottleneck appeared in tasks such as deleting atoms below a target region and expanding a supercell. Switching to a larger model did not produce consistent improvements, and geometric error did not reliably fall with size either. Figure 2 in the study reports both success rate and mean max_dist geometric error. On both the cross-model comparison and the within-Qwen size comparison, the takeaway was similar: scale helps on some rule-bound operations, but not in a stable way once 3D reasoning becomes central.
That is why the paper does not frame AtomWorld as a rejection of Scaling Law. The claim is narrower. Scale can still improve part of the capability stack, but it does not automatically repair the core weakness in 3D physical-space operations. For materials modeling, language reasoning, text-based scientific knowledge and atom-level action cannot be treated as interchangeable.
Knowing the science is not the same as building the structure
The study places this gap against the broader rise of AI for Science. Large models have already shown strong “understanding” in scientific workflows: reading papers, predicting material properties, parsing crystal structures and, in some cases, supporting discovery.
It cites Anthropic’s Claude Science, which breaks research work into auditable pipelines and improved efficiency by 10x in tasks such as review writing and gene analysis. It also points to Google DeepMind’s GNoME, which uses graph neural networks to predict inorganic crystal stability and produced about 2.2 million structures through a closed loop of candidate generation, DFT verification and data feedback.
Those advances helped shape a wider belief that if models can understand materials knowledge, atom construction and structural adjustment should follow naturally. AtomWorld pushes back on that assumption. A model may explain a material’s structural properties and still fail to modify the structure reliably. It may read the periodic table and still fail to execute a stable atom-level operation in 3D space.
The paper adds that this is not only a materials-modeling issue. Scientific research is not a purely textual activity. It is built around hypothesis generation, experiment design, tool use, parameter adjustment, observation, debugging and repeated correction. If AI systems are to participate in science rather than just describe it, they need to do more than explain knowledge. They need to execute actions.
External tools help in limited cases
The industry has often tried to compensate for weak 3D manipulation by connecting models to specialist libraries such as pymatgen. AtomWorld includes controlled comparisons on that front as well.
The result, according to the paper, is limited. External tools improved performance mainly in tasks that involve heavy coordinate calculation, such as atom insertion. In more complex scenarios that require deciding atomic relationships or identifying target spatial regions, the gains were small.
The reason is straightforward. A tool can return precise coordinates, but it cannot make the central decision about where an atom should go or which atoms belong to the target region. If the model itself lacks 3D spatial awareness, the tool may execute the wrong intent more precisely, but the result is still a modeling-logic error.
From Language Scaling to Action Scaling
The paper’s main proposal is not to abandon language scaling, but to rethink what should be scaled next. For AI for Science, the authors argue, Language Scaling remains foundational, yet it is only a starting point for strongly operational tasks such as materials modeling.
What is needed, they write, is “Action Scaling”: scaling up executable action data, decomposed action primitives, simulator feedback, physical-constraint validation and tool-assisted error correction so that models improve not only in language but also in verifiable scientific action.
Current scaling practices are centered on fitting language and knowledge over massive text corpora. The data needed for atom-level modeling is different. Public datasets contain very little high-quality paired supervision linking operation instructions to coordinate changes, which makes it hard for spatial understanding, geometric planning and physically constrained action to emerge from language expansion alone.
In that sense, AtomWorld is more than a benchmark. The researchers describe it as infrastructure for a training loop in materials modeling, with automatically generated tasks, target structures and matching feedback. The broader aim is to move AI for Science away from relying only on larger general-purpose models and toward systems that can build real action capability inside verifiable scientific workflows.
Paper and source details
The work was jointly released by the Suzhou Institute for Advanced Research of the University of Science and Technology of China, the University of New South Wales and other institutions, and was presented at ICML 2026. The paper is at https://arxiv.org/abs/2510.04704, the project homepage is https://masterai-eam.github.io/atomworld/, and the code is at https://github.com/MasterAI-EAM/atomworld.
The original Chinese article cited WeChat account “新智元” and credited the author as LRST.