Research map / Video / Video generation
How Far Is Video Generation from World Model: A Physical Law Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains DiT-based video diffusion models from scratch on a 2D physics simulator to test whether scaling yields physical law discovery, finding ID success, OOD failure, and case-based generalization.
Problem
Whether video generation models trained purely on visual observations can discover classical physical laws and generalize beyond the training distribution, which is needed for them to serve as world models.
Approach
Uses a controlled 2D Box2D/PHYRE simulation testbed with an explicit velocity-extraction tool to evaluate video diffusion models on in-distribution, out-of-distribution, and combinatorial generalization, and probes the retrieval mechanism via attribute-conflict experiments.
Key results
- ID velocity error for DiT-L with 3M data is 0.012, close to ground-truth error of 0.010, but OOD error is 0.427 (uniform motion), an order of magnitude higher.
- On combinatorial generalization with DiT-XL, increasing templates from 6 to 60 reduces the out-of-template human-judged abnormal rate from 67% to 10%.
- Attribute-priority experiments yield the ordering color > size > velocity > shape for case matching, with no exceptions over 1,400 shape test cases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale model size and training data | DiT video generation model · Uniform motion (ID) | velocity error: 0.022 -> 0.012 | Helped |
| Scale model size and training data | DiT video generation model · Uniform motion (OOD) | velocity error: 0.433, 0.328 and 0.358 | No clear effect |
| Increase template coverage from 6 to 60 templates | DiT-XL · Combinatorial out-of-template | abnormal ratio: 67% -> 10% | Helped |
| Use DiT-XL instead of DiT-B with 60 templates | Combinatorial video generation · Combinatorial out-of-template | abnormal ratio: 24% -> 10% | Helped |
| Use 6 templates rather than 60 templates | DiT-XL · Combinatorial in-template | SSIM: 0.972 -> 0.973 | Helped |
| Fine-tune pretrained SVD | SVD · Uniform motion (ID) | velocity error: 0.0138 -> 0.0505 | Hurt |
| Fine-tune pretrained SVD | SVD · Uniform motion (OOD) | velocity error: 0.3583 -> 0.9081 | Hurt |
| Add numeric conditioning to vision (ablation) | DiT-B collision model · Collision (ID) | prediction error | No clear effect |
| Add text conditioning to vision (ablation) | DiT-B collision model · Collision (ID) | prediction error | No clear effect |
| Add numeric conditioning to vision (ablation) | DiT-B collision model · Collision (OOD) | prediction error | Hurt |
| Add text conditioning to vision (ablation) | DiT-B collision model · Collision (OOD) | prediction error | Hurt |
| Replace the primary VAE with the CoGVideo VAE (ablation) | Video diffusion model · Uniform motion attribute comparisons | attribute prioritization | No clear effect |
Surprising findings
- Adding numeric or language conditioning did not improve OOD generalization and language conditioning significantly increased OOD error.
- Fine-tuned SVD performed worse than DiT-B trained from scratch on both ID and OOD errors.
Limitations
- Evaluation limited to simple 2D classical mechanics scenes with geometric shapes, not real-world video.
- Velocity extraction relies on heuristic pixel-based tracking that is unreliable for combinatorial scenes, so combinatorial evaluation depends on human judgment.
- Only DiT-S to DiT-XL and a single SVD fine-tune were tested; findings may not extend to larger or differently trained models.
Experimental setup
Datasets: Box2D-generated 2D kinematics videos (uniform motion, elastic collision, parabolic motion) PHYRE-based combinatorial templates LAION-Aesthetics COYO DataComp Vimeo-90K Panda-70M HDVG
Benchmarks: in-distribution and OOD velocity error on simulated physics scenarios combinatorial in-template and out-of-template test sets
Metrics: velocity error FVD SSIM PSNR LPIPS abnormal ratio (human judged)
Baselines: Groundtruth (parsed GT video) DiT-B SVD-Finetune CogVideo VAE variant
Builds on: Sora DiT PHYRE SD1.5-VAE Box2D
Closest papers by meaning
- PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop (ICML 2025 · similarity 0.90)
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.89)
- Force Prompting: Video Generation Models Can Learn And Generalize Physics-based Control Signals (NeurIPS 2025 · similarity 0.89)
- WorldModelBench: Judging Video Generation Models As World Models (NeurIPS 2025 · similarity 0.88)
- MotionCraft: Physics-Based Zero-Shot Video Generation (NeurIPS 2024 · similarity 0.87)
- VideoPhy: Evaluating Physical Commonsense for Video Generation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.