Researchers from Harvard University, MIT, IBM, Boston University, Google, Johns Hopkins University, Carnegie Mellon University, and the Kempner Institute have introduced MemoBench, a new benchmark aimed at testing object permanence in world models under dynamic occlusion. The paper has been accepted to ECCV 2026 and focuses on a central question for video generation systems: when an object leaves the field of view and continues to change, can a model preserve its identity, infer its updated state, and reconstruct it correctly when it reappears?
The team argues that this capability remains largely unmeasured by current benchmarks. Most existing video generation evaluations emphasize frame-to-frame consistency when an object stays visible. A smaller number include occlusion, but usually in static settings where nothing meaningful changes while the object is hidden. As a result, current test suites do not reliably distinguish between a model that truly maintains an internal state of the world and one that simply generates plausible-looking frames.
The benchmark centers on a visible-disappear-reappear protocol
MemoBench is built around a Visible–Disappear–Reappear, or V-D-R, evaluation framework. In each test clip, the target object is initially visible and undergoing a physical process, such as walking, melting, pouring, burning, or deforming. The camera then moves away through panning, turning, forward motion, or U-turn trajectories until the object leaves the frame entirely. Later, the camera returns to the corresponding region, and the model is expected to render the target in its updated state rather than merely restore its earlier appearance.
According to the authors, this setup directly probes whether a world model maintains a persistent state representation outside the visible frame. In other words, the benchmark is not just about temporal smoothness. It is about whether the model can remember what existed before, continue tracking how it should evolve while unseen, and align the reappearing result with the true physical progression captured in the reference video.
360 ground-truth videos combine synthetic and real-world data
The dataset contains 360 high-quality 1920×1080 ground-truth videos split into two parallel subsets. The synthetic portion includes 196 clips rendered in Unreal Engine 5 across five environment categories and 14 sub-scenes. These clips come with frame-wise RGB images, metric depth maps, camera intrinsics, and camera poses, allowing geometry-aware evaluation under large viewpoint shifts. This part of the benchmark is designed to stress-test spatial memory and camera-grounded scene consistency.
The real-world subset contains 164 clips recorded in controlled indoor environments and covers seven categories and 30 physical state-change processes. These include dissolution, burning and heat-induced transformation, diffusion and absorption, chemical reactions, viscous flow, foam evolution, and mechanical deformation. Because many of these effects depend on material properties such as viscosity, elasticity, and thermal conduction, they are difficult to reproduce faithfully in game engines and therefore provide a stronger test of memory over real physical dynamics.
Every sample is manually annotated to identify the exact frame where the object fully disappears and the exact frame where it fully reappears. These annotations define the visible, disappear, and reappear stages that all subsequent metrics rely on.
Automated scoring is paired with VQA-based semantic evaluation
MemoBench uses two complementary evaluation tracks: automated quantitative metrics and semantic VQA assessment. The automated side covers general video quality, memory-specific performance, and prompt fidelity. The central memory metric is the object reappearance score, or ORS, which uses the text-driven segmentation model SAM-3 to detect whether the target object is present in the reappearance stage and how confidently it can be identified. This is meant to directly capture whether the hidden object comes back correctly.
Other automated metrics include PSNR, SSIM, LPIPS, and camera controllability, helping the authors localize where performance breaks down across the V-D-R stages. Prompt alignment is measured with ImageReward. On the semantic side, the VQA pipeline evaluates four diagnostic dimensions: instruction following, object and background consistency, memory continuity, and physical plausibility. The authors report that after filtering with ground-truth checks, failure-case screening, and human validation, agreement between human annotators and the vision-language evaluation pipeline reached 92.9%, with a Cohen’s kappa of 0.85.
No tested model reliably handles the memory challenge
The benchmark was used to evaluate 10 mainstream world-generation models across three technical families: camera-controllable image-to-video systems, explicit 3D view synthesis models, and standard image-to-video models. The headline finding is straightforward: no tested model reliably solved the disappear-and-reappear memory task. All models scored below 0.6 on ORS, with none demonstrating stable object recovery after off-screen state evolution.
Among the models that actually executed camera motion and caused the target to leave the frame, the highest semantic score for memory continuity was only 55.6 out of 100. LTX-Video reached 57.0, but the authors note that this result is inflated because the model barely moved the camera, keeping the target visible for much of the sequence and sidestepping the benchmark’s central challenge.
The study groups failure modes into six categories: object disappearance, identity drift, state reset to the initial condition, abrupt position jump, background hallucination, and camera trajectory drift. In synthetic scenes, background hallucination was the most common issue, meaning the object might remain but the surrounding world changed incorrectly. In real-world clips, identity drift was more prevalent, where the same object looked significantly different before and after occlusion.
Several comparisons reveal sharp differences across model families
One of the most counterintuitive results came from LTX-Video, a standard image-to-video model without real camera control. It ranked first on motion smoothness, 3D geometric consistency, and object identity consistency, and reached an ORS of 0.330. At first glance, those numbers appear respectable. However, the researchers found that the model frequently kept the camera nearly static, which meant the target object never truly disappeared from view. In practice, its relatively strong consistency scores reflected challenge avoidance rather than actual memory.
MemoBench addresses this by introducing explicit camera controllability and instruction-following dimensions. On those axes, LTX-Video fell into the lowest tier. The result highlights a key blind spot in traditional metrics: they often fail to separate genuine consistency under viewpoint change from superficial consistency created by avoiding camera motion altogether.
The benchmark also shows that camera control does not automatically imply stronger object permanence. Among five camera-controllable image-to-video models, ORS ranged from 0.266 to 0.582, a spread of more than two times. HunyuanWorldPlay led the group with 0.582, while HunyuanGameCraft and FantasyWorld scored only 0.266 and 0.276 respectively, despite posting stronger visual quality scores. This suggests that following a camera trajectory and maintaining a persistent hidden-state representation are independent capabilities.
In the 3D category, the contrast was equally sharp. Stable Virtual Camera delivered the most accurate camera trajectory control and the strongest pixel-level fidelity, but suffered from lower visual quality and visible artifacts such as blurry edges, seams, and depth inpainting errors. Matrix-Game 2.0 showed the opposite profile: sharper visuals and higher aesthetic quality, but frequent camera-path deviation that dragged down geometric and semantic accuracy. The authors argue that geometry precision and visual naturalness remain misaligned in today’s systems, which is why single-metric evaluations can be misleading.
Conditioning on the first visible frame beat simple parameter scaling
The paper also reports an ablation study on Wan2.2. In one setup, the researchers increased model size from 5B to 14B parameters. In another, they kept the model at 5B but added the first frame of the visible stage as an image condition. The latter change delivered a much larger effect: PSNR improved by about 4.2 dB and LPIPS dropped by 0.20, while the near-tripling of parameter count produced only marginal gains.
This result points to a more targeted direction for future work. Rather than relying primarily on larger backbones, the authors suggest that better encoding and preservation of the initial object state may be more important for long-horizon memory tasks. More broadly, the benchmark implies that persistent state representation, memory-centered training objectives, and stronger conditioning on initial observations could become key design priorities for next-generation world models.
Open benchmark aims to become a common yardstick
MemoBench has been released with its paper, code, dataset, and public leaderboard. The paper is titled MemoBench: Benchmarking World Modeling in Dynamically Changing Environments. The arXiv entry is available at https://arxiv.org/abs/2606.27537, the project page is hosted at https://memobench-team.github.io/, and the code is available at https://github.com/MemoBench-Team/MemoBench. The dataset and leaderboard are also publicly accessible through Hugging Face.
For researchers working on video generation, embodied AI, robotics, and world simulation, the benchmark offers a more demanding and more realistic way to test whether a model is doing more than producing visually smooth sequences. MemoBench’s central claim is that there is still a wide gap between generating coherent video and maintaining a faithful internal model of the world, and that gap is now measurable.