NVIDIA Spatial Intelligence Lab has released Lyra 2.0, an open-source framework that generates explorable large-scale 3D worlds from a single image along user-defined camera paths. The output can be exported as 3D Gaussian Splatting or triangle meshes, directly importable into Isaac Sim for robot training.
Two Major Hurdles in Long-Range Generation
Existing video generation models produce visually refined short clips, but quality collapses when extending to full-scene exploration. NVIDIA researchers identified two decay mechanisms: spatial forgetting, where limited context windows cause the model to "hallucinate" revisited areas inconsistently, and temporal drifting, where autoregressive errors accumulate over frames, altering color and texture. The combination renders the "generate video, then reconstruct 3D" approach ineffective for long trajectories.
Lyra 2.0's Two Solutions
To counter spatial forgetting, Lyra 2.0 introduces a spatial memory mechanism. It maintains per-frame 3D geometry, retrieves frames with highest overlap for new viewpoints, aligns them via canonical projection, and injects correspondences into a Diffusion Transformer via attention. Geometry only guides positioning; appearance synthesis relies on generation priors, preserving visual richness without inventing structures.
For temporal drifting, Lyra 2.0 employs self-augmented training: it feeds the model its own noisy historical frames during training, forcing it to correct drift rather than perpetuate errors. The method mirrors students grading their own exams — exposure to mistakes builds automatic correction reflexes.
Interactive Exploration and 3D Export
Lyra 2.0 includes an interactive GUI for real-time point cloud inspection and manual trajectory planning. Scene generation is progressive: the model generates only areas the user explores. After completion, frames are converted to 3D Gaussian Splatting or triangle meshes via feed-forward reconstruction, both ready for physics engines. NVIDIA demonstrated outputting scenes to Isaac Sim for physically grounded navigation and interaction tasks.
Paper: arXiv:2604.13036. Code open-sourced under Apache 2.0 on GitHub; model weights on HuggingFace (nvidia/Lyra-2.0).