SenseTime and a joint team from Xiaoxiao Robot, Ropedia, Nanyang Technological University and the University of Tokyo have released a technical report titled Has GPT-6-Astra Achieved Embodied Intelligence?, offering a broad evaluation of GPT-6-Astra’s embodied capabilities across 12 simulation benchmarks and eight real robot platforms.
The report concludes that GPT-6-Astra has, to a large extent, crossed the threshold for embodied capability. At the same time, it says there is still a meaningful gap between that level of performance and embodied intelligence that can be deployed in practical settings. The experiments surfaced problems tied to cost, latency and unexpected execution failures, which the researchers describe as the next major bottlenecks.
Near-perfect scores on several established benchmarks
Robot manipulation today is commonly built on specialized control policies and Vision-Language-Action, or VLA, models. Those systems are usually trained on real or simulated robot interaction data so they can turn visual observations and task instructions into actions such as grasping, moving and placing.
According to the report, GPT-6-Astra points to a different route. The model can combine visual input, robot state and task requirements to generate action instructions directly, then revise its decisions as execution unfolds. It can also operate across different robot forms.
To study that ability in a systematic way, the team selected 12 representative evaluations spanning five scenario types: tabletop manipulation, dexterous manipulation, humanoid mobile manipulation, long-horizon household tasks and vision-language navigation. GPT-6-Astra was tested under a unified setup in which it read observations, generated robot actions and continued making decisions based on execution results.
The headline scores were strong. GPT-6-Astra reached 100% on LIBERO-Long++, 99.3% on LIBERO-Pro, 98.8% on SIMPLE and 98.9% on LIBERO-Ultra-Long. On more complex dexterous tasks, it posted 87.3% on DexJoCo and 81.9% on RoboDojo. The report says those results matched or exceeded the best publicly available methods at the time on most of the benchmarks.
Two conditions sat behind the high scores
The researchers say those results depended on two key conditions.
The first was in-context learning, or ICL. Before evaluation, GPT-6-Astra was allowed to inspect permitted successful demonstrations from the training set and organize them into a frozen knowledge package containing task descriptions, action trajectories, visual references, helper code and failure recovery methods. Model parameters were not updated, and no new experience was shared across test tasks.
In practice, that meant the model could study prior operating material before entering the test setting. When the robot encountered a scene, it could retrieve similar demonstrations and adapt them to the current object positions. It used prior task experience without retraining model weights.
The second condition was a larger execution budget. Traditional embodied benchmarks often cap the maximum number of action steps. Once that limit is reached, a task is marked as failed even if only a few final actions remain. The team expanded the budget from the standard 1x setting to as much as 10x, allowing the model to keep observing, revising and retrying after failed grasps, object shifts or other execution misses.
That change had a sharp effect on some tasks. On RoboDojo, success rose from 27.6% under the standard budget to 81.9% under the 10x budget. Bench2Dex improved from 25.6% to 61.5%. On tasks that require repeated adjustment of grasp points, bimanual coordination and contact precision, the report says GPT-6-Astra showed a degree of failure recovery.
That improvement did not carry across every benchmark. On the long-horizon household benchmark BEHAVIOR, success was only 37.0%. On the long-horizon navigation benchmark LHPR-VLN, success reached just 28.9% even with the expanded budget. For scenarios that require sustained management of multiple sub-tasks, the benefit of simply allowing more actions was limited.
The model can correct itself, but it still misreads the physical world
The failure cases collected by the team show that GPT-6-Astra can understand a task goal and still fail to judge whether a physical action has actually succeeded.
One example involved hanging a cup on a hook. From the head camera view, GPT-6-Astra judged that the cup handle overlapped with the hook and marked the task as successful. From another camera angle, the cup had not been hung at all and had instead fallen onto the table. The model then tried to re-grasp the cup, but it misread the closing of the gripper as contact with the object, which pushed the rest of the sequence off course.
A similar pattern appeared in a puzzle task. GPT-6-Astra concluded that four pieces had formed the target shape, even though some parts were still misaligned. Because it could not identify what was wrong, it kept adjusting the pieces and ended up disrupting sections that were already close to completion.
The report argues that this is a core difference between physical tasks and language or software tasks. In software, outputs can often be checked through text, program output or test feedback. In robot manipulation, an object that appears to be in the right place may still not be stably contacting, supporting, aligned or fixed. The robot has to judge real contact and spatial relations between objects, and that remains difficult for GPT-6-Astra.
Some physical mistakes are also hard to undo. In a liquid pouring task on RoboDojo, GPT-6-Astra knocked over a cup and spilled liquid onto the table. It then righted the cup and tried to continue pouring, but the spilled portion could not be recovered through a simple retry. In an assembly task on DexJoCo, a part slipped from the robot hand to the floor. GPT-6-Astra judged the part to be outside the operable range and stopped the sequence.
That helps explain why more action steps do not solve every problem. If a grasp is slightly off, the robot can observe again and adjust the gripper. If an object is damaged, liquid is spilled or a part falls somewhere unreachable, more attempts may not help.
Long tasks add another layer. In one example, GPT-6-Astra had already stored a laptop and hung headphones on a stand. It then returned to adjust the headphones, knocked them down and never restored them by the end of the task. The report says that as models get better at planning and correction, verification and recovery during physical execution become the harder problem.
Time and token costs remain heavy
Some benchmarks were nearly solved, but only at striking time and token cost.
On LIBERO-Long++, the standard-budget success rate was already 97.5%, yet each test episode still consumed about 1 million tokens on average, with inference and tool use taking close to one hour. On the longer LIBERO-Ultra-Long benchmark, average token use rose to about 9 million and runtime approached two hours. In the expanded-budget RoboDojo tests, a single episode took more than three hours on average.
Not all of that time went into robot motion. A large share was spent on model inference, checking execution results and calling tools. The harder the task, the more expensive repeated reasoning and retries became.
Harness design changed both success rate and cost
Because repeated retries drove up time and token use, the team explored ways to reduce unnecessary model calls and make each round of reasoning and correction count more. That led them to focus on the Harness, the execution system that links the model, environment observations and robot control interfaces.
On RoboDojo, GPT-6-Astra was used as a System 2 module responsible for reasoning and action review, while a task-finetuned π0.5 VLA acted as System 1 and proposed actions. The team then compared several collaboration modes.
In open-loop collaboration, GPT-6-Astra set subgoals and the VLA generated and executed actions. Success was only 5.83%. In closed-loop collaboration, candidate actions proposed by the VLA were converted into end-effector motions, and GPT-6-Astra decided before execution whether to accept, modify or reject them. Reviewing four candidate actions at a time lifted success to 25.83%. When ICL was added, closed-loop collaboration with two candidate actions per review reached 54.17% success, but token use climbed to about 19.335 million, roughly 6.9 times the closed-loop setup without ICL that reviewed four actions each time.
The Harness also had to solve a second problem: how to translate large-model decisions into robot actions that can be executed accurately.
In direct-control experiments, asking GPT-6-Astra to specify joint angles for the robot arm produced a 9.72% success rate. Switching to end-effector position and pose raised that to 15.83%. Using relative direction and movement distance descriptions pushed it to 18.06%. The report’s takeaway was straightforward: let the large model decide where the arm should move, and leave the detailed joint motion to the low-level controller.
Call frequency mattered too. Re-invoking the model after every step made adjustment easier but was expensive. Executing a fixed multi-step sequence reduced calls but risked missing environmental changes. When the model was allowed to choose whether to execute one to four steps at a time, success improved from 15.83% to 20.00%, while token use fell from 2.56 million to 1.146 million.
Across those experiments, the trade-off was consistent. The large model needs to participate in judgment and correction when necessary, but unnecessary calls have to be cut. How the execution system balances those two goals directly affects both success rate and token cost.
Eight real robot platforms: strong on basic grasping, weaker on fine contact and long-horizon tasks
To test GPT-6-Astra outside simulation, the team deployed it on eight real robot platforms spanning five robot forms. Those included single-arm manipulators such as Franka 3 and Franka Panda, the dual-arm Piper robot, the Tianji M6S with dexterous hands, wheeled mobile manipulation robots and the Unitree G1 humanoid.
Unlike the simulation setup, the real-robot tests did not provide successful task demonstrations to GPT-6-Astra and did not involve extra robot-specific training. The model received structured information written by humans, including robot interfaces, joint limits, camera positions, safety constraints and, on some platforms, operation hints. Execution was measured in real time, usually with a 60-minute limit, and on-site operators supervised safety.
Under those conditions, the same GPT-6-Astra was able to control different robot forms and complete a range of tasks. On rigid-object grasp-and-place tasks, it succeeded in 17 out of 18 tests. On the Unitree G1, using low-level control interfaces, it completed two of three arm-manipulation tasks and all three turn, walk and stop tasks.
Performance dropped once fine contact and long-horizon tasks entered the picture. On a cloth-folding task with Franka 3, it succeeded in two of four attempts. On plug insertion tasks across three robot platforms, it failed all 11 attempts. On long-horizon navigation and search tasks with the AgileX Tracer, it succeeded in nine of 23 tests.
The report also highlighted a flower-arranging task. The robot had to grasp a flower stem and insert it into a vase. With a single-arm Franka 3, the wrist camera could see the stem held by the gripper but not the vase opening below, and the task failed. When the setup switched to the dual-arm Piper, GPT-6-Astra used one arm to hold the flower stem and actively moved the other idle arm so that its wrist camera approached the vase opening. Once the model obtained a clear view, it completed the insertion.
The team described that behavior as a case of active vision. Without task demonstrations, GPT-6-Astra changed its viewpoint on its own to gather the information needed to finish the task.
Real-world tests exposed safety, load and latency issues
The physical environment surfaced problems that simulation does not fully capture.
Safety was one. In some operations, the robot arm triggered protective shutdowns after contacting the table or colliding at the joints, ending the task immediately. Mechanical load was another. During repeated grasp attempts on the Tianji M6S, some finger joints in the dexterous hand stalled because of load issues, and temperature rose to 62 degrees Celsius, interrupting the run.
Decision speed was also a major issue. In Unitree G1 experiments, about 90% of GPT-6-Astra’s runtime was spent on deliberation. A single operation experiment could take more than an hour. While the robot waited for the next action, hardware remained occupied and the time cost, equipment risk and human supervision burden kept rising.
The team argues for a new evaluation style closer to household needs
After seeing GPT-6-Astra approach perfect scores on several embodied benchmarks, the researchers revisited a basic question: how much do those scores really say about a robot’s ability in daily life?
Most current embodied evaluations revolve around clearly specified task goals. Real household tasks are less tidy. The report gives a simple example: if a user says, “Help me prepare breakfast,” and needs to leave in 15 minutes, then having eggs and bread ready within 10 minutes may be more valuable than producing a richer breakfast after an hour. The robot also needs to remember to turn off the stove and close the refrigerator door instead of leaving the kitchen in disorder.
Using a cooking task in BEHAVIOR-1K as an example, the team points to a limitation in some current evaluation rules. If food reaches the required doneness, it may still count as success even if it is burned afterward.
In the team’s view, future embodied evaluations should move closer to actual human needs, allow robots to decide how to complete tasks and account for the value of the final outcome to the user. Tests should not assume ready-made successful demonstrations, and reasoning time should be part of the score.
A household prototype surfaced more execution details
To test those ideas, the researchers built an interactive one-bedroom household environment in NVIDIA Isaac Sim, equipped with a refrigerator, sink, stove and air fryer, while simulating object contact, liquid flow and heating. GPT-6-Astra observed the environment through the cameras and robot state of the simulated AgiBot G2 and attempted tasks such as preparing breakfast, tidying a table and storing items without task demonstrations.
The model exposed a series of issues. While tidying the table, the robot arm knocked over other objects and even pushed the monitor out of place. While storing food, it put milk into the refrigerator and then turned to handle fruit, leaving the refrigerator door open. Faced with an unfamiliar faucet, the robot repeatedly pushed a handle that should have been rotated, failed to shut off the water in time and caused the sink to overflow. While frying an egg, the model judged that the egg was done and decided to turn off the stove, but did not do so until about 90 seconds later. By then the egg had burned, and residual heat kept cooking it.
The report says those household experiments are still small-scale explorations and do not represent deployed performance. Even so, they show that checking only whether a task was eventually completed can miss what happened during execution.
SenseNova-RoboRSI aims to improve the execution system through failure feedback
Looking across the study, the team says GPT-6-Astra’s performance in robot tasks is positively correlated with how much prior experience it can use and how well the model works with the execution system. Task demonstrations, action-interface changes and added feedback checks can all shift success rates sharply without updating model weights.
That observation opens another line of work: how to let robots accumulate experience from past failures and avoid repeating the same mistakes. SenseTime’s answer is SenseNova-RoboRSI, a project that tries to improve the Harness based on task feedback through a unified agent runtime and evaluation framework for robot tasks.
Developers can connect different models, robot control tools, skill modules and execution environments to the same runtime system and inspect the full execution process through real task feedback. The Harness manages what information the model receives, when tools are called, how actions are generated and how the system proceeds after failure.
On top of that, the Harness is meant to keep improving. The report gives a simple example: after a failed grasp, the system analyzes execution traces and feedback, adjusts action generation, planning strategy, feedback checks and memory organization, then returns to the relevant benchmark task for validation. Effective changes are retained in later versions without updating the base model weights. The team links that process to recursive self-improvement, or RSI.
In previously disclosed RoboDojo-Sim experiments, SenseNova-RoboRSI raised the average score from 28.97 to 60.08 and the success rate from 22.48% to 54.17% relative to the GPT-6-Astra baseline.
The report also lists results on other embodied evaluations. On LIBERO-Pro, SenseNova-RoboRSI reached a 94.50% success rate, above the 92.63% posted by the GPT-6-Astra-based RPent method. In the target kitchen setting of RoboCasa365, adding task memory produced a 64.60% success rate, above Paimon-0 at 58.10%.
SenseNova-RoboRSI-27B-Preview explored the effect of combining a smaller model with Harness-based RSI. On RoboDojo, the 27B model with VLA-assisted execution achieved an average score of 26.13, which was 2.84 points below the GPT-6-Astra baseline. On LIBERO-Pro, VLA-assisted execution and direct VLM control reached 95.0% and 96.1% success, both ranking second behind GPT-6-Astra. On RoboCasa365, RSI training lifted success from 26.0% to 46.5% while cutting average output tokens by 73.3%.
The team says the next steps for SenseNova-RoboRSI will focus on the weaknesses exposed in the GPT-6-Astra experiments, including transferring manipulation experience to other objects or robot platforms, using accumulated cases to choose strategies faster and reducing unnecessary large-model calls so reasoning resources are spent on key decisions.
Bottom line: benchmark scores are still short of reliable real-world execution
The report’s final message is that GPT-6-Astra has demonstrated the ability to complete complex operations across robot forms, while RoboRSI shifts attention to self-improving execution systems that learn from failure and refine the Harness. But that route has so far been validated mainly in simulation, and whether those gains will carry over to real robots remains an open question.
High benchmark scores, the researchers argue, are still some distance away from reliable execution in the real world. Users will not accept a robot forgetting to turn off the stove just because it posts strong benchmark numbers. For embodied intelligence to move into daily life, robots still need to show that they can finish tasks steadily and without leaving a mess behind.
Technical report: https://www.sensenova.ai/has-gpt-6-astra-achieved-embodied-intelligence
Project page: https://github.com/OpenSenseNova/SenseNova-RoboRSI
The original Chinese article was credited to the WeChat account Jiqizhixin (ID: almosthuman2014), with the author line shown in the source as “关注具身智能的”.
