Research map / Video / Video generation
WorldSimBench: Towards Video Generation Models as World Simulators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WorldSimBench, a dual evaluation framework (human-preference video scoring via a fine-tuned Flash-VStream evaluator on the HF-Embodied dataset, plus closed-loop action-level tests via video-to-action policies) for video generation models as embodied World Simulators across Minecraft, driving, and robot manipulation.
Problem
Existing predictive-model benchmarks evaluate text plans or aesthetics of images/videos and cannot assess whether generated videos are physically plausible and actionable for embodied tasks, so video generators as World Simulators lack embodied evaluation.
Approach
Classify predictive models into stages S0-S3 and evaluate S3 World Simulators both by a learned human-preference evaluator on fine-grained embodied dimensions (explicit) and by converting generated videos into actions with pretrained video-to-action models in closed-loop simulators (implicit).
Key results
- Human Preference Evaluator reaches 89.4% accuracy on OE versus 72.8% for GPT-4o, and PLCC 0.60 vs 0.28 on AD.
- On RM, the evaluator achieves PLCC 0.43 versus 0.07 for GPT-4o; GPT-4o shows negative correlation with humans on OpenSora in AD zero-shot.
- Closed-loop results show video generators with first-frame conditioning have lower success in Minecraft than text-only models, and Dynamicrafter underperforms Open-Sora-Plan on long-horizon tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Flash-VStream on HF-Embodied instead of using GPT-4o for evaluator scoring | OE human preference evaluator · OE human preference alignment | accuracy: 72.8 -> 89.4 | Helped |
| Fine-tune Flash-VStream on HF-Embodied instead of using GPT-4o for evaluator scoring | AD human preference evaluator · AD human preference alignment | PLCC: 0.28 -> 0.60 | Helped |
| Fine-tune Flash-VStream on HF-Embodied instead of using GPT-4o for evaluator scoring | RM human preference evaluator · RM human preference alignment | PLCC: 0.07 -> 0.43 | Helped |
| Add first-frame image conditioning (ablation) | Open-Sora-Plan in OE closed-loop evaluation · OE travel distance | travel distance: 342.91 to 195.14 | Hurt |
| Add first-frame image conditioning (ablation) | Video generation models in OE closed-loop evaluation · OE embodied tasks | success rate | Hurt |
Surprising findings
- Image-conditioned (first-frame) video models have significantly lower success rates in Minecraft than text-only models.
- Explicit perceptual scores and closed-loop manipulative results do not always agree; e.g., Dynamicrafter ranks high perceptually but underperforms in long-horizon closed-loop tasks.
Limitations
- Evaluation limited to three embodied scenarios and does not cover other physical-world applications.
- Video-to-action intermediaries are task-specific and the closed-loop results depend on them.
- Human Preference Evaluator's zero-shot validation covers only a few generators (OpenSora, Lavie).
Experimental setup
Datasets: HF-Embodied Dataset OpenAI Contractor Gameplay Dataset nuScenes RH20T-P CARLA CALVIN VPT
Benchmarks: HF-Embodied Dataset MineRL Steve-1 tasks LangAuto CARLA benchmark CALVIN ABC-D
Metrics: accuracy PLCC human preference score per dimension success rate driving score route completion travel distance
Baselines: GPT-4o Open-Sora-Plan Lavie ModelScope OpenSora AnimateDiff DynamicCrafter EasyAnimate
Builds on: Flash-VStream Steve-1 LMdrive Dynamicrafter VPT
Closest papers by meaning
- Empowering World Models with Reflection for Embodied Video Prediction (ICML 2025 · similarity 0.92)
- WorldModelBench: Judging Video Generation Models As World Models (NeurIPS 2025 · similarity 0.91)
- 4DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models (CVPR 2026 · similarity 0.89)
- Learning 3D Persistent Embodied World Models (NeurIPS 2025 · similarity 0.89)
- WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors (NeurIPS 2026 · similarity 0.89)
- EA-WM: Event-Aware Generative World Model with Structured Kinematic-to-Visual Action Fields (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.