Scalabot has released HERON-World Model, its first world model. The system takes a current frame plus a specified next action and predicts how the world changes after the robot carries out that action.
The announcement centers on more than single-agent forecasting. Scalabot said HERON-World Model can also predict coordination between multiple agents, the effects of their actions on one another, and the resulting changes in the surrounding world when those actions unfold together.
According to the Scalabot team, HERON-World Model was evaluated offline using the open-source WorldArena 1.0 Track 1 protocol and reached a composite EWMScore_P of 75.80.
Three capabilities shown in the demos
The demos described in the article fall into three groups: physical laws, state memory with temporal consistency, and multi-agent interaction.
The most direct examples involve physical rules. In a tennis-ball throwing scene, HERON-World Model predicts the ball’s flight, descent, collision, and rebound based on the action input.
The second group is state memory and temporal consistency. In the cup-lid example, the model must keep track of which cups have already been covered and where they are located. In a drawing example, it remembers where the pen has already moved and what has already appeared on the canvas. It also keeps the causal chain intact: where the pen tip passes, color remains, and later predictions stay consistent with what has already happened.
The model also fills in information that is not directly visible in the current frame by combining the visual input with the intended action. The article gives the example of a closed refrigerator image followed by the action "open the refrigerator." HERON-World Model then predicts the interior space behind the door.
The third group, and one of the headline features in this release, is multi-agent interaction. Most world models have focused on the "action-state change" chain for a single agent. HERON-World Model extends that prediction range to scenes where multiple agents act together.
That means the model is not limited to asking what happens after one robot moves. It must also account for cooperation or interference between several agents, and for the world state that emerges after those agents and the environment interact at the same time.
The article points to the main difficulty here. Once a task shifts from a single agent to collaborative multi-actor scenes, the number of variables rises sharply. The model has to handle human-object interaction while also understanding what physical outcomes are triggered when different agents’ motions stack on top of one another.
Two examples are given. In one, two people pull a towel at the same time to control the movement of a ball. The model has to keep the relationship among human motion, towel deformation, and ball rolling physically plausible. In another, two people play a string-figure game together. Their actions constantly reshape the string, and the model has to keep later changes consistent with both participants’ movements.
Outside the demos, the article lists category scores from WorldArena. HERON-World Model scored 95.60 in physical interaction, 99.20 in spatial perspective, 92.61 in motion dynamics, and 56.66 in trajectory accuracy. The article says those results validate the model’s performance across physical relations, spatial consistency, motion change, and trajectory prediction.
Why a robot needs to predict the future first
The article frames the value of a world model in practical terms: it lets a robot rehearse the consequences of an action before that action is carried out in the physical world.
Once robots move into real production settings such as factories and homes, they need more than a next-step instruction. They also need a way to anticipate what that next step will cause. A world model’s video generation capability turns that forecast into a continuous sequence of future frames. A robot can preview possible outcomes in the model first, then choose a more suitable action based on those simulated results.
The article also notes that useful prediction depends on learning rich mappings between actions and outcomes from high-quality data. HERON-World Model’s approach is to expose the system to a wide range of worlds through multi-source data, convert actions from different sources into a shared representation, and then use a MoT+MoE architecture to improve future-state rollout.
A three-layer data pyramid: Ego video, simulation, and real robots
The training setup is described as a pretraining data pyramid.
In a conventional embodied-AI data pyramid, internet and human video form the base, simulation sits in the middle, and real-robot data occupies the top. The higher the layer, the closer the data is to real execution, but the harder it is to scale. The article says Ego first-person human operation video has drawn more attention this year because it scales more easily while carrying rich interaction information, and some approaches now place it at the foundation.
HERON-World Model uses three data sources in pretraining: Ego human operation video, simulation data reconstructed from real scenes, and real-robot data.
Real-robot data serves as the "anchor" for actual execution. It records the robot’s visual observations, actions, embodiment state, and control commands at the same time, allowing the model to learn a direct relationship between what the robot did and how the real world changed. According to the article, the dataset covers several embodiments, including single-arm, dual-arm, wheel-arm, and humanoid systems, so the model can observe different forms of movement and interaction across different bodies.
Compared with other data types, real-robot data captures what the robot saw, what action it executed, and how both the robot state and environment changed before and after the action. But the article says this kind of data is expensive, difficult to scale, and poor at covering long-tail cases. It compares the issue to earlier autonomous-driving data collection: some scenarios are rare by nature, some actions are costly to gather, and some failure trajectories can involve equipment damage or safety risks.
That is why HERON-World Model adds simulation data. In the description provided, simulation expands coverage of rare states, dangerous interactions, failure trajectories, and specific scene combinations. It broadens the model’s experience of the action-consequence relationship while keeping real-robot data as the reliable reference point.
Ego data then extends coverage of everyday interaction, adding cases that real robots and simulation struggle to capture, such as tool use, bimanual coordination, and household object manipulation. The article sums the pyramid up this way: real robots provide real action experience, simulation extends unusual situations, and human video broadens interaction coverage.
How heterogeneous data is turned into a shared input format
Once real-robot, simulation, and Ego data are combined, the next problem is formatting them in a way one model can understand.
The article points out that robotic systems can directly log joint angles and control commands, while human video often contains only images. Even when action records exist across datasets, devices still differ in temporal frequency, spatial coordinates, and body structure.
HERON-World Model begins by recovering usable action information from each source. For real-robot and simulation data, actions and states can be extracted from the existing records. For Ego video, the pipeline first cleans the clips, then estimates 3D hand pose, reconstructs camera trajectories, and after trajectory verification and quality filtering, extracts camera motion, hand motion, and grasp state.
This gives human video a trainable action signal. The article adds that these estimated "pseudo-labels" can also be used as weak supervision during training.
From there, the model aligns actions in time and space. Temporally, it uses video frame timestamps as the reference so actions, states, and images are synchronized. Spatially, it converts joint trajectories from different sources into end-effector poses, then uses calibration and coordinate transforms to place movement into a shared frame of reference.
After that step, actions from the three sources are organized into self-motion, manipulator-end motion, and grasp state. The article says this representation no longer depends on the exact joint definition of any one robot. Instead, it uses the first frame of a video clip as the starting point and describes how later states change relative to that point inside a unified action space.
Still, standardizing the representation does not erase embodiment differences. Motions from dual arms, human hands, and different grippers are not easy to unify with rules alone. To address that, HERON-World Model assigns lightweight action encoders to different data domains, maps those actions into fixed-dimensional Action Tokens, and passes them into a shared network that learns the link between action and later visual change.
The article also says HERON-World Model uses masking and confidence control to handle real-world issues such as occlusion and loss of visibility. The goal is to avoid treating "the action cannot be seen" as "there is no action," which helps preserve video samples that still carry training value.
In the article’s summary, action extraction, spatiotemporal alignment, and unified encoding bring all three data sources into the same training flow so the model can learn one central question: after an action is taken, how does the world change?
What MoT and MoE each do in the architecture
To handle scaling at the model level, HERON-World Model uses a combined MoT+MoE design.
MoT is used to keep the model learning action consequences without erasing the Vision-Language Model’s original understanding ability, a point the article refers to as knowledge isolation. The reason, as described there, is that a VLM accumulates broad world knowledge during large-scale pretraining. The article gives examples such as joint constraints on robotic arms and the idea that rigid bodies do not deform for no reason. Those priors can help the model judge whether a predicted future state is plausible.
But later world-model training uses large amounts of data tied to specific manipulations. That can push the system to adapt to new tasks while gradually disturbing or even overwriting the knowledge it already had. HERON-World Model addresses this by separating the parameters of the VLM and the generative branch through MoT, then adding a one-way attention mechanism. In the setup described by the team, the generative branch can read the VLM’s world knowledge while minimizing the chance that generative training feeds back and alters that knowledge. The intended result is that HERON-World Model can use the VLM’s understanding to constrain future-state prediction while reducing forgetting during training.
MoE handles a different issue, one that grows with larger datasets: different actions follow different motion rules. When a robot moves as a whole, global motion and scene structure matter most. During grasping, collision, and contact, local geometry and fine interaction matter more.
This complexity rises again in multi-agent scenes. The model has to capture not only interactions between a single agent and the environment, but also cooperation, conflict, and shared physical changes created when multiple agents act at the same time. If all of that is forced through one parameter set, increasing action diversity can intensify parameter competition between patterns.
HERON-World Model therefore splits the DiT generation process into a high-noise expert and a low-noise expert. The high-noise expert determines the broad future structure and coarse motion first. The low-noise expert then adds local geometry, visual details, and fine interaction.
The article reduces the division of labor to a simple distinction: MoT addresses how knowledge works together, while MoE addresses how generation work is divided. According to the Scalabot team, this hybrid-expert design also leaves room for future scaling by easing parameter competition as datasets grow and action or interaction types become more complex, which supports more stable training and generation.
Looking at the overall technical route, the article ties data, processing, and architecture together in one chain: multi-source data supplies interaction experience, unified handling of heterogeneous actions lets the model learn from that experience, and MoT+MoE supports the generation of future frames. For an action-conditioned world model, the central question remains the same: after a robot takes an action, what happens next?
This article was originally published by the WeChat account QbitAI, written by henry.