Research map / Video / Video generation
EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EnerVerse, a chunk-wise autoregressive multi-view video diffusion model with sparse context memory that predicts embodied future spaces and feeds features into a DiT action head, plus a 4DGS-based data flywheel for sim-to-real.
Problem
Video generation models adapted to robot manipulation ignore the 3D, temporally coherent structure of robotic environments, hurting action prediction, long-horizon consistency, and sim-to-real transfer.
Approach
Autoregressive chunk-wise video diffusion with sparsely sampled clean-frame memory, multi-view ray-direction conditioning pretrained on multi-camera data, and a policy head reading cached first-denoising-step UNet features, combined with a 4D Gaussian Splatting loop to generate training data.
Key results
- On LIBERO, EnerVerse-A with S-RGB input reaches 84.1 average success, and with RGB plus 2 rendered views reaches 88.5, versus 81.2 for MAIL.
- On CALVIN ABC→D, EnerVerse achieves average length 3.00 with S-RGB only, below 3D Diffuser (3.27) which uses RGB-D and proprioception.
- On RT-1 atomic video generation, EnerVerse achieves PSNR 26.1 and FVD 404.65 versus 25.42 and 445.94 for DynamiCrafter (FN).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EnerVerse instead of DC-FN | RT-1 atomic-task video generation · RT-1 Atomic Task | PSNR: 25.42 -> 26.1 | Helped |
| use EnerVerse instead of DC-FN | RT-1 atomic-task video generation · RT-1 Atomic Task | FVD: 445.94 -> 404.65 | Helped |
| use EnerVerse instead of DC-FN | RT-1 video generation · RT-1 Atomic Task | Semantic accuracy: 97 -> 97 | No clear effect |
| add sparse memory (ablation) | EnerVerse policy · LIBERO-Long-SV | success rate: 30.8 -> 73 | Helped |
| train the full model from scratch with policy loss only (ablation) | EnerVerse · LIBERO-Spatial | success rate: Failed | Hurt |
| initialize from DynamicCrafter pretrained weights (ablation) | EnerVerse policy training · LIBERO-Spatial | success rate: Failed -> 79 | Helped |
| co-train policy and video-generation losses (ablation) | EnerVerse policy training · LIBERO-Spatial | success rate: 79 -> 86.3 | Helped |
| use two-stage video pretraining followed by policy fine-tuning (ablation) | EnerVerse policy training · LIBERO-Spatial | success rate: 86.3 -> 92.1 | Helped |
| add multi-view pretraining (ablation) | video-generator policy with single RGB input · LIBERO-Spatial | success rate: 79.0 -> 92.1 | Helped |
| add 1 rendered view at test time (ablation) | EnerVerse-A · LIBERO-Spatial | success rate: 92.1 -> 93.0 | Helped |
| add 2 rendered views instead of 1 (ablation) | EnerVerse policy · LIBERO | average success rate: 85.5 -> 88.5 | Helped |
| skip LIBERO video-generation adaptation (ablation) | EnerVerse policy · LIBERO-Object | success rate: 93.2 -> 85 | Hurt |
Surprising findings
- Training the policy from scratch without pretrained weights failed to converge on LIBERO-Spatial.
- Adding a policy head to a single-view DynamiCrafter yields only 79.0 success versus 92.1 for the multi-view pretrained EnerVerse-A.
Limitations
- Generated videos contain artifacts such as surface penetration and snappy transitions, and pretrained model collapses on LIBERO-Object without adaptation.
- Rendered view poses are heuristically set rather than optimized, and CALVIN evaluation does not reset memory between subtasks.
- Real-world results show failures on some target positions (3,2) and (3,3) and weaker instruction following than OpenVLA.
Experimental setup
Datasets: RT-1 Taco-Play ManiSkill BridgeV1 LanguageTable RoboTurk Isaac Sim multi-view dataset LIBERO CALVIN
Benchmarks: LIBERO CALVIN RT-1 video generation
Metrics: PSNR FVD success rate average success sequence length user study scores
Baselines: Diffusion Policy Octo OpenVLA MDT MAIL RoboFlamingo GR-1 3D Diffuser SUSIE DynamiCrafter (FN)
Builds on: DynamiCrafter Diffusion Policy 4D Gaussian Splatting AVID OpenVLA
Closest papers by meaning
- RoboScape: Physics-informed Embodied World Model (NeurIPS 2025 · similarity 0.89)
- Learning 3D Persistent Embodied World Models (NeurIPS 2025 · similarity 0.89)
- LongScape: Advancing Long-Horizon Embodied World Models with Context-Aware MoE (NeurIPS 2026 · similarity 0.88)
- Empowering World Models with Reflection for Embodied Video Prediction (ICML 2025 · similarity 0.87)
- World Motion Models: Flexible Sequence Modeling of SE(3) Trajectories (NeurIPS 2026 · similarity 0.87)
- WorldSimBench: Towards Video Generation Models as World Simulators (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.