GPT-6 Astra’s viral CAD demos on X helped fuel the idea that frontier models may be getting close to expert-level 3D design work. One widely shared clip showed a turbocharger assembled part by part, able to rotate properly and explode into a disassembled view with a single command. BenchCAD scores added to that impression, with GPT-6 Astra posting 95.9%.
Engineers, though, raised a different point: a model that looks right is not automatically a model that is right. A part that resembles a production component on screen may still fail on scientific principles, geometry, or part relationships.
A benchmark called ScienceVision was built to test exactly that gap. Rather than asking whether a model can imitate the visual surface of CAD software, it checks whether the model can inspect 3D scientific objects the way a domain expert would, verifying structural relations, geometric details, and measurements step by step.
ScienceVision spans five scientific domains and 250 tasks
ScienceVision comes from UniPat AI and the visual reasoning lab Elorian AI. The team said the project started from a simple question: can AI observe and reason like a scientist?
The benchmark goes beyond engineering drawing. It covers biology, chemistry, medical imaging, architecture, and mechanical CAD, for a total of five scientific domains and 22 subfields. The full set contains 250 tasks built from real 3D data, and models must inspect, measure, and analyze those objects through their own actions.
Current results include 12 leading open and closed models, among them GPT-6 Astra, Claude Fable 5.1, Gemini 4 Argon, and Qwen3.8 Max. No model led all five disciplines at once.
The best score so far came from GPT-6-Astra paired with Codex at 0.53 out of 1. In other words, a model that scored 95.9% on BenchCAD reached only about half marks when tested on 3D scientific understanding and reasoning.
What the tasks look like
ScienceVision’s demo shows one inspection sequence from each of the five disciplines. The 3D tools used in those demos were modeled on how experts in each field actually handle source data, with the aim of reproducing realistic observation settings.
- In biology, the full protein is hidden, leaving only a citrate molecule and several nearby ions, and the task is to measure how close they are.
- In chemistry, the shortest sequence takes eight steps: select a unit cell, then apply symmetry operations to see how the structure stacks layer by layer.
- In medical imaging, the model moves through CT slices from three directions to find the boundary of a lesion on the kidney.
- In the CAD task, the object is a forklift, and the model must open parts one by one and count how many share the same shape.
- In architecture, the task is to follow a ramp, measure its length and elevation change, and determine whether the slope exceeds the limit.
The actions themselves are not complicated. Rotate, click, measure. The hard part is deciding where to rotate, what to click, and what to measure before each move. Without solid observation, understanding, and reasoning, the model is unlikely to land on the right answer.
ScienceVision has also opened the original files and inspection tools for those five demo tasks on its blog. Users can switch between 3D and planar views and try the inspection process themselves.
Why the benchmark was built
There have been many 3D understanding and reasoning benchmarks before this, but most focus on indoor scene understanding, embodied interaction, abstract structural reasoning, or other general-purpose tasks tied to everyday perception, robotics, or industrial design.
Benchmarks aimed at scientific 3D understanding and reasoning have been scarce. A few specialized datasets exist in areas such as medical imaging and mechanical engineering, but there has been no unified framework for scientific 3D reasoning across disciplines.
That matters because scientific work depends on observation and reasoning in the real 3D world. Chemists locate atoms and measure geometric relations. Radiologists track structures across slices. Architectural engineers switch viewpoints, remove occlusion, and verify how components connect.
None of that is done by glancing at a single image. Human experts decide where to look, how to look, and what to measure, then apply domain knowledge to reach a conclusion. ScienceVision is trying to test whether AI can do the same.
The benchmark’s logic is similar. The task and the data are presented, and the model must decide for itself where to inspect and how to check. The article compares this to the idea behind Steinmetz’s line: drawing a line may be worth $1, knowing where to draw it is worth $9,999. Reliable observation and understanding come first.
Another problem in earlier scientific 3D evaluation is fragmentation. Different disciplines use different task setups, input formats, rendering logic, and answer forms. ScienceVision is meant to fill that gap with a unified cross-disciplinary benchmark.
250 tasks, 66 experts, 482.98 hours
ScienceVision covers biology, chemistry, medical imaging, mechanical CAD, and architecture. Each discipline contributes 50 tasks, bringing the total to 250 across 22 subfields.
Every task has four parts: a real 3D source file such as molecular coordinates, CT volumetric data, a mechanical assembly, or a building model; a domain question specifying what to inspect, where to find it, and what units or coordinates to use; an answer format; and an expert-labeled reference answer.
Answers are checked with deterministic rules rather than another language model. There is no “close enough.” Either the answer meets the rule or it does not.
The article notes that a benchmark with deterministic verification has another use as well: it can almost directly become a reinforcement learning environment, where correct answers receive rewards and incorrect ones do not, without needing a separate judge model.
Building the benchmark was difficult on several fronts. First, the questions had to be scientifically meaningful and genuinely dependent on 3D evidence. Each one had to start from a concrete scientific object, define the observation target, the decision condition, and the required evidence, and remain solvable in the provided environment.
Second, data from different fields had to remain observable and measurable after conversion and display. Molecular coordinates, CT volumes, mechanical CAD files, and building models differ in format, scale, coordinate conventions, and interaction methods. The system had to preserve key structural information while supporting viewing, isolation, slicing, and measurement across object types.
Third, trustworthy answers and evidence trails had to come from domain experts. Experts checked whether questions were real and relevant, inspected the objects themselves, computed or confirmed the answers, recorded key steps, and took part in review and revision. Because the rules differ across the five fields, that work required field-specific knowledge and experience.
The full pipeline combined automation and human review. An automated system first selected suitable objects from 3D asset libraries, tested whether they could be viewed and measured correctly, generated candidate questions from templates, and filtered them with rules. Experts then took over on a 3D annotation platform, reviewing and revising questions, rotating, isolating, and measuring objects by hand, writing reference answers, recording operation traces, and sending weak items back through quality review. Their feedback was then fed back into the generation pipeline to reduce repeated mistakes.
After repeated cycles of question generation, expert review, automated revision, and pipeline improvement, the benchmark reached its final set of 250 tasks.
UniPat AI built an expert platform spanning the five domains and maintained a pool of 66 specialists. Total expert time reached 482.98 hours: 113 hours in chemistry, 116 in mechanical CAD, 103 in medical imaging, 77 in architecture, and 74 in biology.
Experts reviewed questions, labeled answers, recorded operation processes, and joined multiple rounds of review and revision. That fits the benchmark’s stated goal: if the system is meant to test expert-level observation, the answers themselves need to be grounded in expert work.
Each of the 250 tasks includes operation records like those shown in the demo, with key frames saved step by step. Those records turn scientific observation from a black box that shows only the conclusion into a process that can be replayed, and they may later serve as training targets for models learning how to “look.”
Three evidence settings, one scoring system
ScienceVision compares three ways for models to gather evidence on the same set of 250 tasks.
- In the first setting, the model receives the raw 3D data and writes code to analyze it. The article compares this to being handed the full experimental dataset and doing the calculations yourself.
- In the second, the model gets the raw data plus interactive inspection tools that support viewpoint rotation, structure isolation, slice browsing, and distance and angle measurement. That is the equivalent of getting both the data and a microscope.
- In the third, the model receives only six fixed-angle 2D renders and must infer what it can from those images alone.
All three settings use the same scoring rules, which makes it possible to ask a direct question: do more data and better tools actually improve performance?
Results stayed low, and the reversals were notable
The top score was 0.53 from GPT-6-Astra with Codex, just over half of the full mark. And that figure is not a strict accuracy rate. It is an average metric that includes partial credit, which means the share of fully correct answers is lower still.
The differences across disciplines are more revealing than the overall ranking. No model was best in all five.
- GPT-6 Astra led architecture, mechanical CAD, and chemistry with scores of 0.458, 0.580, and 0.609.
- Claude Fable 5.1 led biology at 0.650.
- Claude Opus 5.5 led medical imaging at 0.399.
Medical imaging produced one of the more surprising patterns. CT is already a stack of slices, and radiologists read slices in routine practice. Models have also seen large amounts of 2D medical imagery during training. But when the input becomes a 3D array with axis order and voxel indices, the models appear much less comfortable.
Scores fell harder when only six fixed-angle renders were provided. In that setup, the best result was just 0.30 from Claude Fable 5.1, while GPT-6 Astra scored 0.25.
Put differently, static pictures were not enough. Raw 3D data did lift scores, but the paper itself cautions that some of those gains may come from code reading coordinates directly rather than from genuine visual understanding.
The more uncomfortable finding is that interactive tools did not help much either. Among the top five models, the largest gain after adding inspection tools was only 0.02, posted by GPT-6.1 Sol. GPT-6 Astra changed by less than 0.01 between the two settings, and Claude Opus 5.5 actually dropped by 0.02.
The paper also notes that even in tool mode, models still read the raw coordinate files. In effect, when given both a microscope and a data sheet, they often chose the data sheet.
Medical imaging stood out again here. Models made roughly three times as many operations as experts, yet posted the lowest scores among the five disciplines. The article’s conclusion is blunt: even with the fullest 3D data and tool support available, current frontier models remain far from expert-level observation and reasoning on real scientific tasks.
More clicks do not mean better observation. They do not mean better reasoning either.
A unified framework for measuring the limits of AI 3D scientific observation
Before ScienceVision, scientific multimodal evaluation did exist, but most benchmarks focused on a single discipline, relied on 2D images, and used their own scoring rules.
The article groups ScienceVision’s value into three points.
First, the tasks are truly 3D. Models receive source files such as molecular coordinates, CT volumes, mechanical assemblies, and building models rather than screenshots. Solving them requires domain knowledge applied to 3D structural evidence, and some tasks require target identification before topology and geometry reasoning can even begin.
Second, the tools mirror expert workflows. View switching, target isolation, slice browsing, and scientific measurement are exposed to models through MCP, with the exact capabilities depending on the viewer. The model decides what to inspect, how to inspect it, and what to measure.
Third, scoring is deterministic and shared across all five disciplines. Each task has a reference answer and preset rules. Components of the answer are checked separately, then aggregated by model configuration and discipline. Because another model is not used as the judge, scores for chemistry molecules and architectural components can be compared directly in one framework.
The article also mentions SciVisAgentBench as the closest related benchmark. It also supports 3D interactive inspection, but the evaluation target is different. ScienceVision uses the same class of domain-constrained structural questions across five disciplines, with structured prompts and per-task reference-answer scoring. SciVisAgentBench is centered more on tasks inside scientific visualization software and evaluates whether an agent’s visualization and analysis outputs meet task requirements.
The article’s bottom line is that ScienceVision is the first benchmark to measure, within a unified framework, the current boundary of AI capability in 3D scientific observation. Knowing where the gap is comes before closing it.
Its closing line is direct: the next time someone says the AI scientist has arrived, let it finish these 250 tasks first.
The article was originally published by the WeChat public account Xinzhiyuan and credited to ASI Qishilu.
