A team led by Tsinghua University has released HandsOn, a compositional simulation platform and benchmark built to evaluate dexterous manipulation across different robot embodiments. The project was developed with researchers from Beijing University of Posts and Telecommunications, Peking University, and Nature Will. It includes 66 manually validated tasks and 18,000 human teleoperated demonstrations in simulation, with evaluation centered on tool use, fine contact, bimanual coordination, and transfer.
The work starts from a gap the authors say has become harder to ignore. Earlier robot manipulation benchmarks were largely built around grippers. As first-person human manipulation data has grown, more detailed records of how people grasp objects, use tools, and manage delicate contact have become available. Dexterous hands, with structures closer to human hands, are seen as a better fit for learning from that data at scale. But the benchmarks have not kept pace with the hardware and models.
Why the team says dexterous hands need a new benchmark
The paper argues that dexterous manipulation is not simply a matter of adding more fingers than a gripper has. The harder part is controlling changing contact relationships throughout an interaction. After picking up a mouse, for example, a robot has to avoid clicking too early. When setting two knobs, it has to keep both at their target states at the same time. When placing a box lid, it must preserve alignment after release.
Those requirements change how success should be measured. For grippers, a benchmark can often rely on whether an object was grasped or moved to a target location. For dexterous hands, that is not enough. A system may need to use a tool correctly, complete a fine manipulation step, and preserve the right object state across multiple stages. Progress in one motion does not mean the full task is done.
The authors say growing stores of egocentric human manipulation data create a new opening for dexterous-hand learning, but two questions remain central: can a model reliably reproduce operations it has seen, and can it transfer those skills to new objects, new task compositions, or deformable items? Existing benchmarks already cover different robot embodiments, functional manipulation, and bimanual coordination, but the team says there is still room to improve the scale of human demonstrations, the use of held-out tasks for evaluation, and the diagnosis of where failures happen.
How HandsOn is organized
HandsOn separates execution from generalization by splitting its 66 tasks into two sets: Core-50 and Transfer-16. Core-50 provides demonstrations for policy learning and then evaluates the learned policy through a fresh round of execution tests. Transfer-16 is reserved for generalization and does not include demonstrations of the target tasks in the training set.
The platform supports five robot arms, six dexterous hands, and about 200 assets spanning rigid objects, articulated objects, and deformable objects. These can be combined in randomized scenes. The benchmark package is structured so that Core-50 includes demonstration content, while Transfer-16 is used only for evaluation.
Core-50: can a model execute tasks it has been shown?
Core-50 contains 50 tasks with training demonstrations. The tasks cover functional understanding, numerical input, spatial relations, articulated-object manipulation, precise contact, and bimanual coordination. The examples in the article are concrete and household-like. A robot manipulating scissors must lift them off the table and complete three open-close cycles while maintaining the grasp. Setting a timer requires entering the specified minutes and seconds, then withdrawing both hands after activation. Opening a folding fan requires both hands to hold the outer ribs, spread the fan to a target angle, and keep it there. Making coffee chains together several subgoals, including taking a cup, closing a door, receiving coffee, and placing the cup.
The task design follows three dimensions. The first is interaction type: basic manipulation, functional use, and contact-intensive manipulation. The second is task structure: single-stage tasks, repeated operations, and long-horizon tasks. The third is object type: rigid objects, objects with rotational or sliding joints, and deformable objects. Within Core-50, the three interaction categories contain 17, 18, and 15 tasks respectively.
Transfer-16: can learned skills move to new settings?
Transfer-16 contains 16 evaluation tasks with no target-task demonstrations provided during training. It is meant to test whether a model can carry over what it has learned. Seven tasks focus on asset or relation transfer, six on skill composition, and three on deformable-object manipulation.
Asset or relation transfer asks a model to identify the right object or target relation in a new setting and complete the task. Skill composition asks it to connect previously learned actions in the correct order. The deformable-object tasks introduce items such as towels, sponges, and pillows, where the object shape changes during manipulation.
One example in the article is a task where the robot must hand over a block between two hands, place it into a box, then align and close the lid. Each local action may have appeared elsewhere in training, but the model still has to decide whether the current step is complete and whether the result of the previous step has been preserved. A more difficult example is repeated towel folding, where the robot must complete two folds in sequence and leave the towel stably on the table after release.
Task cards and stage tracking
To make the evaluation reliable, the team created a task card for every task. Each card specifies the goal, the standard initial state, the expected final state, allowed and forbidden actions, and the evidence used to decide whether the task is complete. Completion can be judged from object poses, joint states, or image-based signals. The researchers also carried out manual validation and added an independent review process to check whether success labels matched the actual physical state.
For multi-step tasks, the benchmark records ordered stage progress. In a block handover task, for instance, it distinguishes between both hands contacting the object and the receiving hand being able to support it independently. In skill-composition tasks, the evaluation checks whether prerequisite steps were completed and whether earlier results were maintained. That makes it possible to tell apart total failure from partial completion followed by a later breakdown.
18,000 demonstrations and how they were collected
Unlike gripper datasets, dexterous-hand demonstrations need finger-level detail. HandsOn uses human teleoperation to convert human hand motions into dexterous-hand action demonstrations. Across the benchmark, the team collected about 56 hours of human operation data from at least 10 operators over more than 1,050 collection sessions. Operators wore MANUS data gloves and used wrist-tracking devices to control the simulated robot. The system recorded images, states, actions, and language information at the same time.
Of the 18,000 demonstrations, 15,178 are used for training and 2,822 for validation. Every interaction includes three image streams: a frontal view and left and right wrist views. The frontal camera captures the tabletop layout and both hands, while the wrist cameras provide close-up observations near the fingers and manipulated objects. For tasks that depend on precise contact, those local views add information about how fingers approach, grasp, and operate an object.
What happens as demonstration counts rise
The team also ran single-task experiments with π0.5 on two tasks: placing a ball into a bowl, and lifting scissors and opening-closing them. Training demonstrations were increased step by step to 10, 50, 100, and 200, then to the full training pool. The complete pools contained 438 and 439 demonstrations respectively. Each checkpoint was trained for 20,000 updates and tested over 100 episodes. The shaded intervals in the figure are 95% Wilson intervals. The article notes that the experiment used a single training seed, and the intervals do not include uncertainty from training-seed choice or subset selection.
Success on the ball task rose from 7% to 83%. Success on the scissors task rose from 1% to 44%. Both improved steadily across the five training scales. The comparison also shows a persistent gap between the two tasks, which the authors use to illustrate the added difficulty of combining stable grasping with tool-oriented functional motion.
HandsOn can also expand the visual side of the data. The platform supports high-fidelity re-rendering of the same interaction while changing tabletop materials, lighting, room backgrounds, and small camera offsets, all while preserving the original state and action supervision. That allows researchers to generate multiple observations from the same interaction and test how a policy handles appearance changes.
Simulation setup and real-robot correspondence
On the hardware side, HandsOn builds a simulation counterpart around a real ROKAE robot arm and the first-generation Wuji Hand, with 20 active joints in each hand. Kinematics, joint limits, asset scales, camera calibration, and state and action coordinates are all recorded in the correspondence. A main camera and wrist cameras are used together so that scene layout and hand-level detail enter the same data and evaluation pipeline.
The broader platform supports five robot arms, six dexterous hands, and about 200 assets, while the data collection and quantitative evaluation reported here use the ROKAE-Wuji and PhysX configuration. The team also released videos of real-robot runs using policies adapted from simulation demonstrations. Those videos qualitatively show contact, transport, and pose maintenance without target-task real-world demonstrations. The article adds that the evidentiary scope of those real-world videos is different from the quantitative simulation evaluation.
Results across 10 methods
HandsOn evaluates 10 methods in total: five Vision-Language-Action models, or VLAs; three World Action Models, or WAMs; a task-specific Diffusion Policy; and GPT-6 as a zero-shot control reference. The first nine methods were trained on HandsOn demonstrations, and eight of them used multitask training.
On Core-50, the nine trained methods completed nearly 45,000 valid test runs. Among multitask policies, OpenWAM posted the highest average task success rate at 29.1%. Performance varied sharply by task. Grasping, transport, and some articulated-object operations were relatively easier, while using a tool on a third object and carrying out precise coordinated actions remained difficult.
The transfer results were much harsher. On Transfer-16, the average task success rate for all eight trained multitask policies stayed below 1%. The single-task Diffusion Policy was not included in that part of the evaluation. The authors say this shows how limited current methods still are when they have to turn prior experience into full-task success on new assets, composed skills, and deformable objects.
Where bimanual systems break down
Because full successes are rare, stage-level progress offers a clearer view of what goes wrong. The dual-knob task is one example. Each hand operates one knob, both sides must reach their target states, and the result has to satisfy hold-and-release conditions. Across 800 attempts from eight multitask models, 219 runs got each side to its target at some point, 38 got both there at the same time, and only 5 satisfied the full task conditions. The gap, the team writes, comes from making two local results hold together. A common failure mode is that the hand that reaches its target first keeps turning, so the earlier result changes before the other hand finishes.
The block handover task shows the timing problem between two hands. Out of 367 episodes where both hands held the block at the same time, 188 reached the point where the receiving hand had taken over independently. Of the remaining 179, 175 ended with the block being dropped. The release timing of the giving hand has to match the support state of the receiving hand.
Long-horizon tasks add ordering constraints. In a composed task that requires closing a kettle first and then closing a box, 340 of 400 attempts moved the box lid first. The article frames these records as traceable research questions: how a model decides that the current step is complete, how it preserves earlier results, and when it should start the next action.
The study also checked whether actions changed with language targets by keeping the scene fixed and replacing the target object in the instruction. Some checkpoints showed limited responsiveness to object names, suggesting that target selection itself still needs close evaluation. A failed task may reflect contact-control problems, but it may also begin earlier with target understanding or action selection.
GPT-6 zero-shot control
GPT-6 provides a separate reference point. In each round, it reads three RGB image streams, robot state, task instruction, and execution feedback, then controls the system through the move_eef interface by specifying palm pose and up to 40 finger-joint targets. Without HandsOn demonstration training or fine-tuning, the model was tested 10 times on each of the 66 tasks, for 660 trials in total.
It recorded 26 successes on Core-50 and 3 on Transfer-16, corresponding to success rates of 5.2% and 1.9%. The article says the action logs show that the model can sometimes propose plans tied to manipulation prerequisites. In one towel-folding attempt, for example, it suggested lifting slightly first to confirm that the towel edge moved with the hand before continuing the fold. That episode still failed, which the authors use to illustrate the difficulty of turning a plan into contact and motion control.
Paper, code, and dataset are public
The paper, code, and dataset have all been released. The paper is on arXiv, the code is on GitHub, and the dataset is on Hugging Face.
- Paper (arXiv): https://arxiv.org/abs/2610.26765
- Code (GitHub): https://github.com/TsinghuaC3I/HandsOn
- Dataset (Hugging Face): https://huggingface.co/datasets/Cyan666/HandsOn-Datasets
The original Chinese article was published by the WeChat account Jiqizhixin (机器之心, ID: almosthuman2014) and credited to Chen Chen and Clio.