Research map / Video / Video generation
WorldModelBench: Judging Video Generation Models As World Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WorldModelBench, a benchmark with 67K human labels that scores video generation models as world models on instruction following, commonsense, and physics adherence, plus a fine-tuned 2B VLM judge used as a reward for video diffusion fine-tuning.
Problem
Existing video generation benchmarks focus on general visual quality and miss world-modeling failures such as physics violations and instruction non-compliance, so it is unclear whether video generators can act as reliable world models.
Approach
Score videos on fine-grained instruction-following levels (0-3), five binary physics laws, and commonsense criteria across 7 application domains; fine-tune a 2B VLM judge on large-scale human labels and use its differentiable answer-token probability gap as a reward (VADER-style) to improve a video diffusion model.
Key results
- Fine-tuned 2B judge reports 4.1% average prediction error on total scores, and it beats zero-shot GPT-4o on the instruction, commonsense and physics error rates (e.g. physics 24.0% vs 36.0%).
- Top closed model KLING scores 8.82 of 10 on human labels, while the best open model Mochi and OpenSoraPlan-T2V score about 7.6.
- Correlation of physics-adherence win rates between WorldModelBench and VBench is only 0.28, versus 0.69 for frame-wise quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use I2V instead of T2V | CogVideoX · WorldModelBench | total score: 7.31 versus 6.75 | Hurt |
| Use I2V instead of T2V | OpenSora-Plan · WorldModelBench | total score: 7.62 versus 6.62 | Hurt |
| Use I2V instead of T2V | OpenSora · WorldModelBench | total score: 6.11 versus 5.83 | Hurt |
| Fine-tune the 2B judger on human annotations | VILA-2B world-model judger · WorldModelBench test set | average accuracy predicting world-modeling violations: 8.6% higher average accuracy | Helped |
| Optimize OpenSora-T2V using rewards from the fine-tuned judger | OpenSora-v1.2 T2V · WorldModelBench prompts | world-modeling capability | Helped |
Surprising findings
- Luma has higher frame-wise and temporal quality than the best open model Mochi but much worse instruction following (44% vs 53% of videos finishing the task).
- I2V models score lower than their T2V counterparts across all three model pairs.
Limitations
- Small benchmark of 350 prompts (WorldModelBench-Hard 45 prompts), limiting statistical power.
- Reward-based improvement is only qualitatively shown on OpenSora-v1.2 T2V with no quantitative gains reported.
- Judge trained on a fixed set of 14 models' outputs per prompt, so generalization to unseen models is not established.
Experimental setup
Datasets: WorldModelBench human annotations reference videos from driving, robotics, human activity datasets Kinetics-style general video source (unspecified)
Benchmarks: WorldModelBench WorldModelBench-Hard VBench
Metrics: average score (instruction 0-3, commonsense 0-2, physics 0-5 prediction error vs human pairwise win rate ELO pairwise agreement Kendall tau correlation of win rates
Baselines: GPT-4o Gemini-1.5-Pro VBench VideoPhy Qwen2-VL-2B
Builds on: VADER VBench VideoPhy Bradley-Terry model chain-of-thought
Closest papers by meaning
- WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors (NeurIPS 2026 · similarity 0.92)
- WorldSimBench: Towards Video Generation Models as World Simulators (ICML 2025 · similarity 0.91)
- Trimming the Long-Tail of Visual World Modeling Evaluation (NeurIPS 2026 · similarity 0.90)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.89)
- Empowering World Models with Reflection for Embodied Video Prediction (ICML 2025 · similarity 0.89)
- VBench: Comprehensive Benchmark Suite for Video Generative Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.