Research map / Video / Video generation
WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WorldReasonBench (436 cases, 4 dimensions, 22 subcategories) and WorldRewardBench (~6K expert preference pairs over 1.4K videos) to test video generators as world-state predictors with process-aware QA verification and multi-dimensional quality scoring.
Problem
Existing video generation benchmarks emphasize perceptual quality or prompt alignment and do not test whether a generator can predict the correct future world-state evolution from an initial image and action, nor do they provide calibrated expert preference data for reward-model evaluation.
Approach
Reframes video generation evaluation as world-state prediction, combining structured ground-truth QA with phase-level process-aware scoring (Score_PR) and a 1-5 multi-dimensional VLM judge, validated against expert Elo rankings and preference pairs.
Key results
- Seedance2.0 achieves the top overall Score_PR of 39.8 among 11 generators, with closed-source models at 32.4-39.8 versus open-source at 14.4-17.9.
- Score_PR achieves Spearman rho 0.955 with human Elo, above the pairwise VLM-judge Elo (rho 0.804).
- Qwen3.5-9B-Thinking reaches 74.35% pair-wise agreement without ties on WorldRewardBench, the best judge among those tested.
Surprising findings
- Pairwise VLM judge places Sora2 models above Seedance2.0, contrary to human ranking, attributed to frame-budget effects.
- Explicit textual hints help open-source generators substantially more (+56-85% relative QA gain) than Sora2 (+29%).
Limitations
- Judge accuracy drops on close pairs, especially Seedance2.0 vs Sora2 family.
- Fixed 8-frame budget in pairwise judging biases results toward longer clips.
- Information-Based dimension remains hardest for both generators and judges.
Experimental setup
Datasets: WorldReasonBench WorldRewardBench
Benchmarks: WorldReasonBench WorldRewardBench
Metrics: Score_PR Acc_QA S(v) pairwise agreement Spearman rho point-wise accuracy Elo
Baselines: GPT-5.4 Gemini-3.1-Flash Qwen3.5-9B Qwen3.5-27B Sora2 Kling Wan2.6 Seedance2.0 Veo3.1-Fast LTX2.3 Wan2.2-14B UniVideo
Builds on: VBench-2.0 V-ReasonBench VIPER Gen-ViRe VideoVerse WorldSimBench
Closest papers by meaning
- WorldModelBench: Judging Video Generation Models As World Models (NeurIPS 2025 · similarity 0.92)
- Learning World Models for Interactive Video Generation (NeurIPS 2025 · similarity 0.89)
- WorldSimBench: Towards Video Generation Models as World Simulators (ICML 2025 · similarity 0.89)
- Empowering World Models with Reflection for Embodied Video Prediction (ICML 2025 · similarity 0.88)
- MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos (ICLR 2025 · similarity 0.88)
- Impossible Videos (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.