Research map / Video / Video generation
Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PhyGenBench (160 prompts over 27 physical laws in four domains) and PhyGenEval, a hierarchical VLM/LLM-based evaluator, to test whether text-to-video models generate physically plausible videos.
Problem
Text-to-video models are judged mainly on visual quality and spatial relations, not on whether generated videos follow basic physical commonsense (intuitive physics), and existing metrics such as FVD and VLM judges are poor at detecting physics violations.
Approach
A benchmark of physics prompts tied to explicit physical laws, paired with a three-stage evaluator (key phenomena detection on keyframes, physics order verification across multi-image inputs, and overall naturalness scoring) where GPT-4o generates questions and evaluation standards, and VLMs answer them.
Key results
- PhyGenEval reaches overall Spearman's rho 0.81 and Kendall's tau 0.78 with human physical-commonsense scores, versus 0.19 rho for VideoScore and 0.18 for DEVIL.
- Best-performing model Gen-3 scores only 0.51 average PCA on PhyGenBench, with Kling close behind at 0.49.
- Scaling up models and prompt engineering were found insufficient to fix dynamic physical phenomena in the discussion analysis.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PhyGenEval instead of VideoScore for physical commonsense evaluation | Evaluation on PhyGenBench · PhyGenBench overall | Spearman correlation with human judgments: 0.19 -> 0.81 | Helped |
| Increase model size from 2B to 5B | CogVideoX · PhyGenBench | PCA average: 0.39 -> 0.45 | Helped |
| Apply the two-stage semantic evaluation strategy (ablation) | Grid-LLaVA evaluation on PhyGenBench · PhyGenBench overall | Spearman correlation with human judgments: 0.39 -> 0.44 | Helped |
| Use GPT-4o instead of Grid-LLaVA for semantic evaluation (ablation) | PhyGenEval · PhyGenBench overall | Spearman correlation with human judgments: 0.44 -> 0.56 | Helped |
| Replace the two-stage video evaluation strategy with a one-stage shared scoring template (ablation) | PhyGenEval video-stage naturalness evaluation · PhyGenBench | Correlation with human judgments | Hurt |
| Use all three physical-evaluation stages together rather than only one or two stages (ablation) | PhyGenEval · PhyGenBench | Correlation with human judgments | Helped |
| Ensemble the evaluation methods instead of using only open-source models (ablation) | PhyGenEval physical commonsense evaluation · PhyGenBench | Correlation with human judgments | Helped |
| Scale up the T2V model | T2V models evaluated on PhyGenBench · PhyGenBench dynamic physical phenomena | Physical commonsense correctness | Mixed |
| Apply prompt engineering | T2V models evaluated on PhyGenBench · PhyGenBench | Physical commonsense correctness | Mixed |
Surprising findings
- Optics prompts were handled better than other categories across all models, while mechanics, thermal, and material prompts were weaker.
- Open-source LLaVA-Interleave and InternVideo2 ensemble alone reached a correlation of 0.66, so GPT-4o is not strictly required.
Limitations
- Benchmark is small with 160 prompts across 27 laws.
- Evaluation relies on GPT-4o and other VLM judges whose physics understanding is imperfect.
- Human evaluation covered only 64 prompts across 8 models.
Experimental setup
Datasets: PhyGenBench prompts (160)
Benchmarks: PhyGenBench
Metrics: Kendall's tau Spearman's rho PCA score SA score
Baselines: VideoPhy VideoScore DEVIL Grid-LLaVA FVD
Builds on: VQAScore DEVIL T2V-CompBench VideoPhy VideoScore
Closest papers by meaning
- VideoPhy: Evaluating Physical Commonsense for Video Generation (ICLR 2025 · similarity 0.89)
- PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation (CVPR 2025 · similarity 0.88)
- Apple-$\pi$: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence (NeurIPS 2026 · similarity 0.87)
- WorldSimBench: Towards Video Generation Models as World Simulators (ICML 2025 · similarity 0.87)
- WorldModelBench: Judging Video Generation Models As World Models (NeurIPS 2025 · similarity 0.86)
- MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.