Research map / 3D vision / 3D generation
Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PhysFlow, which uses GPT-4 to infer material types and initial parameters for 3D Gaussian splats, then refines parameters via optical-flow loss through video diffusion and differentiable MLS-MPM for 4D physical simulation.
Problem
Simulating dynamic scenes requires capturing diverse material properties and object interactions, but prior methods are limited to few material types (mostly elastic), need manual parameter initialization, or require multi-view inputs; render and SDS losses are unstable or memory-heavy for large mot
Approach
Combine GPT-4 physics reasoning for material type/parameter initialization with 3D Gaussian splat reconstruction, and optimize material parameters with an optical-flow loss computed on video-diffusion-generated videos through a differentiable MLS-MPM, instead of render or SDS losses.
Key results
- Human evaluation on real-world scenes: physical-realism 3.44 vs 2.67 (PhysGaussian) and 2.58 (PhysDreamer); only method above 3.
- ECMS on real-world scenes: 3.08 for PhysFlow vs 3.70 (PhysGaussian), 3.87 (PhysDreamer), 4.41 (Physics3D) (lower is better).
- Synthetic system identification: average RE 0.29 and EPE 0.51 for flow loss vs 0.41/1.33 for render loss and 0.70/1.90 for SDS loss.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use optical flow loss instead of render loss (ablation) | PhysFlow · Synthetic dataset (average) | EPE: 1.33 -> 0.51 | Helped |
| use optical flow loss instead of SDS loss (ablation) | PhysFlow · Synthetic dataset (average) | EPE: 1.90 -> 0.51 | Helped |
| initialize material properties with GPT and optimize them (ablation) | PhysFlow · Real-world dataset | ECMS: 3.91 -> 3.08 | Helped |
| optimize material parameters after GPT initialization (ablation) | PhysFlow · Real-world dataset | ECMS: 3.23 -> 3.08 | Helped |
| use optical flow guidance instead of render loss (ablation) | PhysFlow · Real-world dataset | ECMS: 3.21 -> 3.08 | Helped |
Surprising findings
- Manually tuned or further-optimized parameters can yield overly rigid results, while GPT-4-initialized values without optimization already give a more balanced starting point.
Limitations
- System identification remains challenging for complex materials such as Cream and Trophy, where errors remain comparable to baselines rather than clearly better.
- Real-world evaluation relies on a small human study and qualitative comparisons with 36 videos; no standard quantitative benchmark for real-world physics.
- Dependence on video diffusion model outputs, which may be unstable or noisy for large motions.
Experimental setup
Datasets: PAC-NeRF synthetic dataset PhysDreamer real-world scenes Instant-NGP Fox NeRFStudio Plane Mip-NeRF 360 Kitchen self-collected Jam and Sandcastle
Benchmarks: PAC-NeRF synthetic dataset real-world scenes from PhysDreamer and others
Metrics: relative error (RE) end-point error (EPE) human Likert ratings for physical-realism and photo-realism Energy-Constrained Motion Score (ECMS)
Baselines: PAC-NeRF GIC PhysGaussian PhysDreamer Physics3D Lrender SDS loss
Builds on: PhysGaussian PhysDreamer PAC-NeRF GIC Physics3D PhysGen Grid4D
Closest papers by meaning
- DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors (AAAI 2025 · similarity 0.88)
- OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation (ICLR 2025 · similarity 0.86)
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.85)
- Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (ICLR 2024 · similarity 0.85)
- Vid2Sim: Generalizable, Video-based Reconstruction of Appearance, Geometry and Physics for Mesh-free Simulation (CVPR 2025 · similarity 0.85)
- Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.