Research map / Video / Video generation
VideoPhy: Evaluating Physical Commonsense for Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VideoPhy, a 688-caption benchmark for physical commonsense and semantic adherence in text-to-video generation, human-evaluates 12 T2V models, and trains VideoPhy-Eval (a fine-tuned VideoCon) as an automatic evaluator.
Problem
Text-to-video models are increasingly claimed to be physical world simulators, but it is unclear whether generated videos follow physical commonsense and match the text prompt; existing benchmarks like VBench do not measure physical law adherence.
Approach
A benchmark of 688 human-verified captions covering solid-solid, solid-fluid, and fluid-fluid interactions, scored by humans on binary semantic adherence and physical commonsense, plus a fine-tuned video-language auto-rater (VideoPhy-Eval) that is multi-task trained on human annotations.
Key results
- CogVideoX-5B achieves SA=1 and PC=1 on 39.6% of test prompts in human evaluation; all other models are below 20%.
- VideoPhy-Eval reaches ROC-AUC 82 (SA) and 73 (PC), versus 65/54 for zero-shot VideoCon and 73/58 for Gemini-1.5-Pro-Vision.
- On unseen generative models, VideoPhy-Eval reaches 79 SA and 72 PC ROC-AUC versus 64/57 for VideoCon.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale CogVideoX from 2B to 5B | CogVideoX · VideoPhy test prompts | SA, PC (%): 18.6 -> 39.6 | Helped |
| Use CogVideoX-5B instead of the other evaluated video generators | VideoPhy benchmark · VideoPhy test prompts | SA, PC (%): 39.6; other models <20% | Helped |
| Use Dream Machine rather than CogVideoX-5B | Text-to-video generation · VideoPhy test prompts | SA (%): 61.9 vs 63.3 | No clear effect |
| Use Dream Machine rather than CogVideoX-5B | Text-to-video generation · VideoPhy test prompts | PC (%): 53 -> 21.8 | Hurt |
| Generate videos for harder rather than easier captions | Text-to-video models · VideoPhy easy vs hard captions | SA and PC | Mixed |
| Train VideoPhy auto-rater on generated videos and human annotations | VideoCon · VideoPhy test prompts, automatic rating | ROC-AUC SA: 65 -> 82 | Helped |
| Train VideoPhy auto-rater on generated videos and human annotations | VideoCon · VideoPhy test prompts, automatic rating | ROC-AUC PC: 54 -> 73 | Helped |
| Train VideoPhy auto-rater on selected models and evaluate on unseen generative models (ablation) | VideoCon · VideoPhy test prompts, unseen generative models | SA: 64 -> 79 | Helped |
| Train VideoPhy auto-rater on selected models and evaluate on unseen generative models (ablation) | VideoCon · VideoPhy test prompts, unseen generative models | PC: 57 -> 72 | Helped |
| Fine-tune Lumiere-T2I2V on VideoPhy training pairs labeled SA=1 and PC=1 (ablation) | Lumiere-T2I2V · VideoPhy test prompts | SA: 46 -> 36.5 | Hurt |
| Fine-tune Lumiere-T2I2V on VideoPhy training pairs labeled SA=1 and PC=1 (ablation) | Lumiere-T2I2V · VideoPhy test prompts | PC: 25 -> 24.6 | No clear effect |
| Train separate classifiers for semantic adherence and physical commonsense instead of one multitask classifier (ablation) | VideoPhy auto-rater · VideoPhy automatic evaluation | evaluation performance | No clear effect |
Surprising findings
- Dream Machine has high semantic adherence (61.9%) but low physical commonsense (21.8%), suggesting optimizing SA does not imply PC.
- Fine-tuning Lumiere-T2I2V on high-scoring benchmark videos decreased semantic adherence (46 to 36.5) with no PC gain.
Limitations
- Human annotators are mostly US/Canada AMT workers, so judgments reflect Western perceptual biases.
- Benchmark is small (688 captions) and covers only solid and fluid physics, not e.g. projective geometry.
- Automatic evaluator underperforms on physical commonsense (PC ROC-AUC 73) and is weaker on models like Pika in the leaderboard.
Experimental setup
Datasets: VideoPhy benchmark captions human annotations on generated videos
Benchmarks: VideoPhy
Metrics: semantic adherence rate physical commonsense rate joint SA=1 and PC=1 rate ROC-AUC
Baselines: GPT-4V Gemini-1.5-Pro-Vision VideoCon CogVideoX-2B VideoCrafter2 LaVIE SVD-T2I2V ZeroScope OpenSora Pika Dream Machine Lumiere-T2V
Builds on: VideoCon VBench GPT-4
Closest papers by meaning
- Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (ICML 2025 · similarity 0.89)
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.89)
- ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation (NeurIPS 2024 · similarity 0.88)
- PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop (ICML 2025 · similarity 0.88)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.88)
- How Far Is Video Generation from World Model: A Physical Law Perspective (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.