Research map / Video / Video segmentation and tracking
Seeing the Wind from a Falling Leaf
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Recovers invisible force fields (e.g., wind) from videos by backpropagating through a differentiable MPM physics simulator over 3D Gaussians, using a causal tri-plane force representation and a sparse 4D keypoint tracking objective.
Problem
Estimating invisible, spatially varying external forces (e.g., wind) from visual observations of object motion in video, which is harder than estimating a few scalar physical parameters and whose gradient-based optimization through physics simulators is unstable.
Approach
An end-to-end differentiable inverse-graphics pipeline that couples 3D Gaussians (Lagrangian elements) with an MPM simulator and an Eulerian causal tri-plane force field, optimized with sparse keypoint tracks (barycentrically interpolated to Gaussians) instead of dense flow or photometric losses; VLM commonsense supplies material properties.
Key results
- On synthetic scenarios, the method recovers ground-truth force fields with low magnitude and direction errors (exact numbers in tables not in extracted text).
- GPT-4o material-type classification achieves F1 of 1.0 on a 17-object benchmark, with 4.65% MAPE on density.
- Sparse tracking loss and causal tri-plane representation are reported to outperform image reconstruction loss, Flow+Depth loss, and K-planes/point contact forces.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a causal tri-plane force representation instead of K-planes or point contact forces (ablation) | Force recovery framework · Synthetic scenarios | force recovery performance | Helped |
| Use sparse tracking loss instead of image reconstruction loss or Flow+Depth loss (ablation) | Force recovery framework · Synthetic scenarios | force accuracy | Helped |
| Estimate material type and physical properties from a single image with GPT-4o-Vision | VLM material-property estimation · 17 everyday objects | material classification F1: 1.0 | Helped |
| Estimate density from a single image with GPT-4o-Vision | VLM material-property estimation · 17 everyday objects | density MAPE: 4.65% | Helped |
| Estimate Young's modulus from a single image with GPT-4o-Vision | VLM material-property estimation · 17 everyday objects | Young's modulus log-MAPE: 7.17% | Helped |
Surprising findings
- Photometric loss alone was found to give insufficient motion constraints, often yielding vanishing gradients.
Limitations
- Applies mainly to objects with small or bending-only deformations.
- Fluids and other object types requiring different differentiable physics are out of scope.
- Foreground physics is composited over a static background rather than fully modeled scene.
Experimental setup
Datasets: synthetic objects rendered from physics simulator Internet videos real force-gauge experiment videos MatWeb material database
Benchmarks: synthetic MPM scenarios (elastic, elastoplastic, viscoplastic)
Metrics: PSNR SSIM LPIPS force magnitude error (%) force direction error (degrees) F1 MAPE log-MAPE
Baselines: K-planes image reconstruction loss Flow+Depth loss point contact forces
Builds on: 3D Gaussian splatting material point method K-planes CoTracker
Closest papers by meaning
- Force Prompting: Video Generation Models Can Learn And Generalize Physics-based Control Signals (NeurIPS 2025 · similarity 0.87)
- MonoPhysics: Estimating Geometry, Appearance, and Physical Parameters from Monocular Videos (NeurIPS 2026 · similarity 0.85)
- How Far Is Video Generation from World Model: A Physical Law Perspective (ICML 2025 · similarity 0.84)
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.84)
- PhysGen3D: Crafting a Miniature Interactive World from a Single Image (CVPR 2025 · similarity 0.84)
- DecoupledGaussian: Object-Scene Decoupling for Physics-Based Interaction (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.