Research map / Video / Video generation
WISA: World simulator assistant for physics-aware text-to-video generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WISA, a framework that decomposes physical principles into textual, qualitative, and quantitative physical information and injects them into CogVideoX-5B via Mixture-of-Physical-Experts Attention and a Physical Classifier, plus a 32K-clip physics video dataset WISA-32K.
Problem
Text-to-video models generate videos that fail to follow real-world physical laws, partly because abstract physical principles are not explicitly conveyed to the generator and existing datasets weakly represent specific physical phenomena.
Approach
Decompose physical principles into textual physical descriptions, qualitative physics categories, and quantitative physical properties, and inject each through a different route: caption concatenation, expert-head attention gated by category vectors (MoPA) with AdaLN for quantities, and a auxiliary physical classifier loss.
Key results
- On VideoPhy, WISA reaches SA 0.67 and PC 0.38, versus 0.60 and 0.33 for the CogVideoX-5B baseline.
- On PhyGenBench, WISA reaches PC 0.43 (best) and SA 0.40 (second), with inference time 220 s versus 1800 s for PhyT2V.
- Ablation: removing the Physical Module drops SA to 0.64 and PC to 0.33; using a 32K Koala-36M subset instead of WISA-32K gives SA 0.62 and PC 0.33.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add WISA's structured physical guidance and components to CogVideoX-5B | CogVideoX-5B · VideoPhy | SA: 0.60 -> 0.67 | Helped |
| add WISA's structured physical guidance and components to CogVideoX-5B | CogVideoX-5B · VideoPhy | PC: 0.33 -> 0.38 | Helped |
| add the Physical Module (ablation) | WISA · VideoPhy | SA: 0.64 -> 0.67 | Helped |
| add the Physical Module (ablation) | WISA · VideoPhy | PC: 0.33 -> 0.38 | Helped |
| add the Physical Classifier (ablation) | WISA · VideoPhy | SA: 0.66 -> 0.67 | Helped |
| add the Physical Classifier (ablation) | WISA · VideoPhy | PC: 0.36 -> 0.38 | Helped |
| train with only LoRA (ablation) | CogVideoX-5B · VideoPhy | PC: 0.33 -> 0.34 | Helped |
| replace WISA-32K with 32K labeled samples from Koala-36M (ablation) | WISA · VideoPhy | PC: 0.38 -> 0.33 | Hurt |
Surprising findings
- Human and VideoCon-Physics scores do not fully align; a physically correct sample received only 0.08 from VideoCon-Physics, and Qwen2.5-VL also failed to judge event order.
- Training on 32K general-scene Koala-36M clips with the same physical labels gives only limited improvement, versus the physics-focused WISA-32K.
Limitations
- Covers only 17 physical phenomena and omits others such as corrosion or vacuum settings.
- Provides high-level semantic guidance only, lacking mechanism-level constraints such as energy conservation or Newton's laws.
- Automatic physics evaluator (VideoCon-Physics) shows misjudgments, and the model still fails in many scenarios.
Experimental setup
Datasets: WISA-32K Koala-36M (32K subset, ablation) VideoPhy prompts PhyGenBench prompts
Benchmarks: VideoPhy PhyGenBench
Metrics: SA PC human preference ranking inference time
Baselines: VideoCrafter2 HunyuanVideo CogVideoX-5B Cosmos-Diffusion-7B PhyT2V
Builds on: CogVideoX MoH Mixture-of-Experts AdaLN PhyT2V
Closest papers by meaning
- VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models (NeurIPS 2025 · similarity 0.89)
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.88)
- PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation (CVPR 2025 · similarity 0.88)
- Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation (NeurIPS 2025 · similarity 0.87)
- How Far Is Video Generation from World Model: A Physical Law Perspective (ICML 2025 · similarity 0.87)
- Force Prompting: Video Generation Models Can Learn And Generalize Physics-based Control Signals (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.