Research map / Video / Video generation
DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns a KAN-based material field from video diffusion priors via motion distillation sampling, then drives 3D Gaussian Splatting with an MPM physics simulator to generate realistic 4D dynamics.
Problem
Animating static 3D scenes with realistic motion: manually set physics parameters yield unnatural results, while video-diffusion-distilled deformation fields produce small, discontinuous motion.
Approach
Learn physical material properties (not deformation) from video diffusion priors through a differentiable MPM simulator, using motion distillation sampling that subtracts the first-frame-view prediction to suppress color bias, plus a KAN-based tri-plane material field and frame boosting for stable optimization.
Key results
- On 4 scenes, Ours achieves aesthetic score 5.03 vs PhysGaussian 4.98, PhysDreamer 4.84, DreamGaussian4D 4.61 (GT 5.13).
- Ablation: full model reaches 29.71 average iterations vs 36.86 for the baseline, with aesthetic score 4.93 vs 4.86.
- User study: Ours preferred in 55.36% of SOTA comparison votes vs 21.43% for PhysDreamer and 16.07% for PhysGaussian.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DreamPhysics instead of PhysGaussian | 4 comparison scenes · 4 comparison scenes | aesthetic quality: 4.98 -> 5.03 | Helped |
| Use DreamPhysics instead of PhysDreamer | 4 comparison scenes · 4 comparison scenes | aesthetic quality: 4.84 -> 5.03 | Helped |
| Add a KAN-based material field (ablation) | SDS-T baseline · 7 scenes | aesthetic quality: 4.86 -> 4.89 | Helped |
| Add a KAN-based material field (ablation) | SDS-T baseline · 7 scenes | training iterations: 36.86 -> 34.29 | Helped |
| Add motion distillation sampling (ablation) | SDS-T + KAN · 7 scenes | aesthetic quality: 4.89 -> 4.94 | Helped |
| Add motion distillation sampling (ablation) | SDS-T + KAN · 7 scenes | training iterations: 34.29 -> 33.86 | Helped |
| Add frame boosting to KAN + MDS (ablation) | SDS-T + KAN + MDS · 7 scenes | aesthetic quality: 4.94 -> 4.93 | No clear effect |
| Add frame boosting to KAN + MDS (ablation) | SDS-T + KAN + MDS · 7 scenes | training iterations: 33.86 -> 29.71 | Helped |
| Replace KAN layers with an MLP encoder in the tri-plane material field (ablation) | DreamPhysics · Ball collision | shape preservation | Hurt |
Surprising findings
- Pre-generated videos from video generators can be worse than manual physics settings as supervision, as PhysDreamer scored lower than PhysGaussian.
- VBench motion smoothness scores above 0.99 for all physics-based methods, so the authors judged it unable to discriminate physical realism.
Limitations
- Simulated motions depend on independent physical constraints; hard to combine multiple motion types in one simulator.
- Simulators handle only a few target objects; environment interactions such as shadows are not modeled.
- Evaluation relies on aesthetic score and a small user study on seven scenes.
Experimental setup
Datasets: PhysDreamer scenes (Alocasia, Carnation, Hat, Telephone) PhysGaussian scenes (Ficus, Pillow) LGM-generated Ball
Benchmarks: VBench aesthetic quality on 4 scenes (vs GT) ablation on 7 scenes user study
Metrics: aesthetic quality score (LAION predictor via VBench) average training iterations user preference percentage
Baselines: PhysGaussian PhysDreamer DreamGaussian4D
Builds on: score distillation sampling PhysGaussian variational score distillation Kolmogorov-Arnold networks 3D Gaussian Splatting
Closest papers by meaning
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.89)
- Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulation (CVPR 2025 · similarity 0.88)
- OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation (ICLR 2025 · similarity 0.88)
- Latent Intuitive Physics: Learning to Transfer Hidden Physics from A 3D Video (ICLR 2024 · similarity 0.88)
- MotionCraft: Physics-Based Zero-Shot Video Generation (NeurIPS 2024 · similarity 0.87)
- FreeGave: 3D Physics Learning from Dynamic Videos by Gaussian Velocity (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.