Research map / Video / Video generation
Physics-Informed Video Generation via Mixture-of-Experts Latent Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PILA, a frozen-flow-matching video adapter that builds a physics attribute bank from generator latents, routes top-k physics-category experts with label priors, applies operational PDE-style residual regularization, and adds a correction to the velocity field, reporting gains on VideoPhy-2, VBench-2.0 and PhyGenBench.
Problem
Pretrained video generators produce visually convincing videos that often violate physical regularities (collisions, gravity, deformation, interactions), and simulator-based fixes need costly 3D geometry and states.
Approach
Construct an operational physical attribute bank (field-proxy slots) from frozen-generator latents using observable motion as a kinematic anchor, route category-specific operator experts via prompt-derived label priors, regularize them with category-specific operational residuals, and decode the refined bank into an additive correction to the frozen flow-matching vector field.
Key results
- Ours-1.3B reaches Joint 0.740 and Rule 0.965 on VideoPhy-2 vs 0.643 and 0.852 for Wan 2.1-1.3B.
- Ours-14B (direct adapter transfer to Wan 2.2-14B) reaches PhyGenBench average 0.683 vs 0.538 for Wan 2.2-14B.
- Ablation on Wan 2.1-1.3B: removing flow correction drops PhyGenBench average from 0.615 to 0.507, the largest degradation among ablations.
Surprising findings
- Removing the flow correction is worse than removing the whole MoE operator or routing, suggesting the bank's benefit depends mainly on modulating the vector field.
- Imaging quality of the 1.3B model is roughly unchanged from baseline (0.600 vs 0.589 on VBench imaging in one table, 0.543 vs 0.538 in ablation), so physics gains come with little visual cost.
Limitations
- Residual constraints are operational proxies rather than full governing equations or calibrated physical states.
- Relies on LLM-assisted prompt parser for label priors at inference.
- 14B results come from transferring an adapter trained on Wan 2.1-1.3B without 14B-specific training.
Experimental setup
Datasets: WISA-80K VideoPhy PhyGenBench
Benchmarks: VideoPhy-2 VBench-2.0 VBench PhyGenBench
Metrics: Joint score Rule score Dynamic Mechanics Material Motion Imaging quality PhyGenBench Optics/Thermal/Average
Baselines: Wan 2.1-1.3B CogVideoX-5B VideoREPA-2B Hunyuan Video Wan 2.2-14B WISA-14B
Builds on: Wan 2.1 VideoREPA WISA VideoPhy-2
Closest papers by meaning
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.88)
- VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models (NeurIPS 2025 · similarity 0.88)
- LaMo: Self-Supervised Latent Motion Priors for Physical Realism in Video Generation (NeurIPS 2026 · similarity 0.88)
- Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation (NeurIPS 2025 · similarity 0.87)
- VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models (ICML 2025 · similarity 0.87)
- WISA: World simulator assistant for physics-aware text-to-video generation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.