Video generation models have advanced quickly in image clarity, temporal consistency, and controllability. Given text, images, camera trajectories, or player actions, they can now produce coherent follow-up frames, pushing the idea of an interactive video world from a concept toward viewable and operable prototypes.
But continuing frames is not the same as running a world. Motion and collisions on screen are only the most visible layer of interaction. In more complex environments, characters carry goals, identities, and relationships. Events are constrained by rules and leave consequences that can stretch over much longer spans. Some state changes even happen off-camera and still affect what happens later. Predicting the next observation from local pixels alone makes it hard to explicitly maintain that higher-level semantics.
The paper argues that a complex world first needs to answer why it changes the way it does. Goals, rules, memory, and causal chains usually cannot be read straight from a single frame, yet they determine what should happen next. Language models, with their ability to call on knowledge, reason, plan, and write code, are better suited to those low-frequency but high-complexity decisions and can turn them into executable rules.
Another point raised by the researchers is that game video is not the world itself. It is a pixel projection rendered from a program’s execution. At this stage, games and simulators remain the main source of interactive data for video world models. If one keeps only actions and pixels, then asks a video model to learn the mapping between the two, that effectively bypasses the underlying program, rules, and explicit world state. The model is then forced to infer the full program behavior backward from visual output. The paper says that path is inefficient and entangles world evolution with visual presentation inside the same model.
From that observation, teams from Westlake University AGI Lab and Nanyang Technological University proposed Code World Model, a new world-model framework with a language model as the “brain.” The core idea is to let a coding agent continuously write, call, and modify code to maintain and update an executable world state. States relevant to the current observation are then converted into a proxy that guides a video model in generating the final visual output.
The paper is available at https://arxiv.org/abs/2608.25927. The project page is https://buaacyw.github.io/cwm/, and the code repository is https://github.com/buaacyw/code-world-model.
How the framework divides responsibilities
In this setup, the language model does not replace the video model. Code World Model redraws the boundary between them. Knowledge, goals, rules, and long-range causality are handled by the coding agent and code. Appearance, motion details, lighting, and texture are handled by the video model. One side decides what happens in the world. The other decides what that looks like.
Figure 1 in the paper summarizes the split: the coding agent evolves world state through code, the proxy turns relevant state into visual conditions, and the video model produces the final frames.
Method overview: coding agent, code, and video model
The paper breaks world execution into three linked parts.
- Coding agent: understands player intent and new events, draws on world knowledge, reasons about possible consequences, and decides which mechanisms need to be executed, combined, or modified.
- Code: runs deterministic updates such as position, attributes, collisions, cooldowns, and event triggers at higher frequency, turning high-level decisions into rules that can be inspected, reused, and kept running.
- Video model: reads the evolved state conditions and uses priors learned from large-scale visual data to generate observations with texture, lighting, and local motion.
This separation also maps to different compute frequencies. The coding agent only needs to make sparse decisions when a new goal appears, a complex event occurs, or existing mechanisms are not enough. Once that decision is written into code, the program can keep executing dense state updates.
More than that, the coding agent can modify not just values at one moment, but the way the world will operate afterward.
Figure 2 shows the overall framework. The full vision includes visual feedback, while the current prototype focuses on validating the forward chain of coding agent / code → world state → proxy → video model.
Proxy as the interface between executable state and generation
After responsibilities are split, one interface problem remains. The coding agent and code maintain executable state, while the video model takes text, image, or video tokens. The two sides need a common language that is easy for programs to construct and still provides frame-by-frame spatial constraints.
Structured text is flexible, but it is hard to use it to stably describe exact entity positions, relative relations, trajectories, and camera motion for every frame. A fully built and rendered 3D world offers stronger control, but it brings back the high cost of assets, geometry, materials, animation, and rendering pipelines.
To solve that, the researchers introduce the proxy. It extracts only the information from world state that the current observation must follow and organizes it as a coarse visual representation. A lightweight, deterministic compiler renders that proxy into proxy video, which is then sent to the video model together with structured text.
Text says who is involved, what things are, and what should happen. The proxy specifies where things are, how they move, and how the camera should look at them.
The proxy can represent camera and viewpoint, entity position and orientation, scale and trajectory, scene layout and occlusion, as well as coarse interaction state needed for the current observation. Texture, materials, fine lighting, and full local motion are intentionally left to the video model. The paper summarizes the rule this way: keep only the minimum sufficient state needed for the current observation, balancing controllability against the cost of state encoding.
In the current implementation, the proxy has only one quarter of the target video’s width and height, or roughly one-sixteenth of its pixel count. It is built from simple, reusable geometric primitives. It does not require production-quality assets, but it still provides an editable spatiotemporal skeleton frame by frame.
Strictly aligned data from gameplay runtime logs
For the video model to learn the proxy, each training example needs proxy video, structured text, and target RGB video, with strict alignment across time, camera, and entity identity.
Gameplay runtime logs naturally preserve that correspondence. The team synchronously records RGB, camera state, entity identity, position and orientation, approximate scale, scene layout, and interaction state from the same gameplay execution, then compiles the proxy offline through code. Because both the RGB output and the proxy come from the same execution, a one-to-one mapping can be established for every frame, and identity can be tracked across frames.
Figure 3a shows target RGB from game data on top and the matching proxy below. Figure 3b shows target RGB from real video on top and an offline-constructed proxy below, built using camera calibration, 3D reconstruction, and object annotation.
The gameplay dataset used in the paper contains 157 recordings with about 5.6 hours of source video. Sampling at 2-second intervals produced 9,420 training clips of 5 seconds each. Every RGB target contains 124 frames at 1344×768 and 24 FPS. The matching proxy also contains 124 frames at 336×192, combined with fixed-log depth and a categorical semantic-ID map.
The same runtime record can also be recompiled into proxies with different coverage and information granularity, without recording RGB again.
The paper also demonstrates an offline pipeline for constructing proxy-observation pairs from real video on KITTI-360, showing how the interface could extend to real-world data. The current video-model adaptation still uses only paired gameplay data.
Prototype setup in training and inference
The current prototype uses the Ref2VA backbone of MiniMax-H3 as the video model. It applies rank-128 LoRA adaptation to all 50 transformer blocks, with about 596 million trainable parameters. Training ran on 8 NVIDIA H800 GPUs for 3 epochs and 3,534 optimization steps.
At inference time, GPT-5.6 Sol serves as the coding agent. The system provides it with basic player controls, collision handling, a runtime update loop, existing game-engine scene and gameplay logic templates, and the proxy primitives used during training. The coding agent can combine, extend, and rewrite that code to construct simple executable worlds that a player can control. The coarse 3D geometry in this setup is used only to express the proxy. It is not the final rendered output.
For each 5-second clip, GPT Image 2 generates an appearance anchor from the first-frame proxy and text. MiniMax-H3 then uses the first frame, the full proxy sequence, and text to generate a 124-frame RGB video at 1344×768 and 24 FPS.
Longer videos are produced with a sliding window that overlaps by 34 frames. A later window inherits RGB context from the end of the previous one and reuses the same appearance anchor to preserve local continuity and overall identity appearance.
What the experiments showed
The qualitative results in the paper show that even with LoRA adaptation on only about 5.6 hours of gameplay source video, the model can still follow the positions, motion trajectories, scene layout, and camera movement specified by the proxy when characters, environments, and style differ clearly from the training appearance. At the same time, it fills in texture, lighting, and local dynamics.
Figure 4 shows the proxy providing character positions and action trajectories, while the video model renders that same coarse skeleton into characters with different identities and art styles. Figure 5 shows how simple geometric primitives constrain complex objects and camera motion. In both sets, the proxy appears on top and the generated result below.
One example in the paper says that the same coarse human primitive can be rendered as characters with completely different identities and artistic styles. Wireframes or boxes can constrain the position and broad motion of vehicles, boats, and other complex objects. The same proxy interface can also express running, dancing, swimming, falling, and orbiting camera motion.
That means the proxy fixes the spatiotemporal skeleton that the current observation must follow, not the assets or visual style of the training game.
The project page also includes video comparisons with action-conditioned or camera-conditioned video world models. According to the article, the comparison is about control granularity and visual results: the proxy directly provides frame-level entity motion and viewpoint changes, allowing finer and more direct spatiotemporal constraints. The paper does not present that comparison as an inference-latency benchmark.
Tim Sweeney replies to speculation about Unreal Engine
One commenter wrote: 「Will Unreal Engine 6 be the last engine to create games with traditional 3D methods? Maybe Unreal Engine 7 will use a similar technical route. Only Unreal Engine CEO Tim knows the answer.」
Tim then replied: 「I don’t know either!」
The paper’s closing argument
The article says past video world models mainly learned how an observation should continue. Code World Model pushes the problem one step earlier: before generating the next observation, it first maintains what the world currently is, which rules it follows, and why the consequence of an event persists.
According to the researchers, a truly open generative world needs both the visual imagination of a video model and an execution mechanism that can maintain knowledge, goals, rules, relationships, and causal consequences. Their core message is simple: let code decide what happens in the world, and let the video model decide what that looks like.
The reference cited is https://arxiv.org/abs/2608.25927. The original article was published via the WeChat account Xinzhiyuan, written by Xinzhiyuan and edited by LRST.