Research map / Video / Video generation
Empowering World Models with Reflection for Embodied Video Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Reflection-of-Generation (RoG), a VLM-driven self-reflection loop for chunk-wise autoregressive video prediction, instantiated as the EVA world model and evaluated on the new EVA-Bench.
Problem
Video generation world models for embodied scenarios struggle with multi-step prediction, knowing when a task is complete, and out-of-distribution scenarios.
Approach
Couples a pre-trained VLM (understanding: task-completion reflection and text) with a video diffusion model (generation) and iteratively extends or regenerates video chunks based on the VLM's self-check, trained in three stages with domain-specific Ensemble-LoRA and a generation adapter.
Key results
- EVA reaches a GPT-4o score of 62.63 on Action-Description, versus 29.58 for the best zero-shot open model (Qwen2).
- On Finish-Thinking video generation, EVA achieves GCE 89.09 and FVD 184.81, versus DynamiCrafter's GCE 80.96 and FVD 362.56.
- On RT-1 in-domain tasks, EVA succeeds on 42/50 versus 29/50 for AVDC; in OOD, 12/15 versus 7/15.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the VLM on EVA instruction data and use EVA’s multistage training | EVA · Action-Description | GPT-4o score: 38.46 -> 62.63 | Helped |
| Fine-tune Dynamicrafter on EVA-Instruct as EVA-Gen | Dynamicrafter · Finish-Thinking | GCE: 80.96 -> 86.83 | Helped |
| Add the EVA understanding module and RoG to the generation model | EVA-Gen · Finish-Thinking | FVD: 177.28 -> 184.81 | Hurt |
| Use EVA instead of EVA-2Stage | EVA-2Stage · How-To | EVA-Score: 79.42 -> 81.67 | Helped |
| Use EVA instead of EVA-2Stage | EVA-2Stage · Next-Step | EVA-Score: 68.74 -> 70.85 | Helped |
| Add RoG to EVA (ablation) | EVA · RT-1 in-domain tasks | successes: 41/50 -> 42/50 | Helped |
| Add RoG to EVA (ablation) | EVA · RT-1 OOD tasks | successes: 5/15 -> 12/15 | Helped |
| Add RoG to EVA (ablation) | EVA · CALVIN video task completion | successes (lightbulb): 12/41 -> 35/41 | Helped |
| Add RoG to EVA (ablation) | EVA · CALVIN video task completion | successes (open drawer): 18/41 -> 36/41 | Helped |
| Replace CMLP with CMLP2 (ablation) | EVA · CALVIN simulation task completion | successes (lightbulb): 9/41 -> 32/41 | Helped |
| Add RoG to EVA with CMLP2 (ablation) | EVA + CMLP2 · CALVIN simulation task completion | successes (lightbulb): 10/41 -> 32/41 | Helped |
Surprising findings
- Removing the RoG reflection loop lowers success on the OOD RT-1 subset (w/RoG 5/15 for AVDC-based setting versus 7/15 baseline), suggesting reflection does not uniformly help in every configuration.
Limitations
- Evaluation benchmark is small (125 samples), and OOD robot tasks are few.
- Open/Close tasks with transparent glass doors remain challenging.
- Reported EVA-Score for the main model is inconsistent (81.67 in Table 3 vs 81.64 in text).
Experimental setup
Datasets: EVA-Instruct COCO CC3M-595K VideoChatGPT Open-X-Embodiment Ego4D Ego-Exo4D CALVIN RoboVQA
Benchmarks: EVA-Bench RT-1 CALVIN
Metrics: BLEU-1 METEOR ROUGE-L CIDEr SPICE CLIP score GPT-4o score Subject Consistency Background Consistency Motion Smoothness
Baselines: DynamiCrafter AVDC GPT-4o Qwen2 LLaVA-OneVision LLaVA-NeXT ChatUniVi MiniCPM-V2
Builds on: DynamiCrafter ChatUniVi LoRA AVDC RoboDreamer
Closest papers by meaning
- WorldSimBench: Towards Video Generation Models as World Simulators (ICML 2025 · similarity 0.92)
- WorldModelBench: Judging Video Generation Models As World Models (NeurIPS 2025 · similarity 0.89)
- Learning 3D Persistent Embodied World Models (NeurIPS 2025 · similarity 0.89)
- WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors (NeurIPS 2026 · similarity 0.88)
- Learning World Models for Interactive Video Generation (NeurIPS 2025 · similarity 0.88)
- LongScape: Advancing Long-Horizon Embodied World Models with Context-Aware MoE (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.