Research map / 3D vision / Face and shape
Vid2Sim: Generalizable, Video-based Reconstruction of Appearance, Geometry and Physics for Mesh-free Simulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Vid2Sim reconstructs appearance, geometry, and elastic physical properties from multi-view video via a feed-forward VideoMAE+LGM predictor followed by a few-minute refinement with a mesh-free LBS reduced simulator and a neural Jacobian.
Problem
Recovering textured geometry and physical parameters (e.g., Young's modulus, Poisson's ratio) from video for simulation, which prior methods do via slow per-scene optimization with grid-based MPM or limited mass-spring models.
Approach
Combine a generalizable feed-forward physics predictor (VideoMAE backbone with hypernetwork-predicted LBS weights, trained on Objaverse simulations) with a short scene-specific refinement using an implicit-Euler mesh-free reduced simulator and a learned Neural Jacobian to speed up deformation gradient computation.
Key results
- Mean PSNR 30.17 on 12 synthetic GSO scenes vs 22.06 (PAC-NeRF), 20.91 (Spring-Gaus), 21.01 (GIC).
- Future state prediction mean PSNR 25.07 vs 20.11 for PAC-NeRF.
- Total refinement time about 15 minutes per scene vs 54 to 120 minutes for baselines on an RTX 4090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Vid2Sim instead of the strongest baseline | Synthetic dynamic reconstruction · Synthetic dynamic reconstruction (12-object mean) | PSNR: 22.06 -> 30.17 | Helped |
| use Vid2Sim instead of Spring-Gaus | Real-world reconstruction · Real-world objects (mean) | PSNR: 26.05 -> 27.12 | Helped |
| use Vid2Sim instead of the strongest baseline | Synthetic future state prediction · Synthetic future state prediction (mean) | PSNR: 20.11 -> 25.07 | Helped |
| add scene-specific refinement after Stage I (ablation) | Vid2Sim · Synthetic dynamic reconstruction (12-object mean) | PSNR: 19.77 -> 30.17 | Helped |
| initialize 3D Gaussians from LGM rather than training them from scratch (ablation) | Vid2Sim · Synthetic dynamic reconstruction (12-object mean) | PSNR: 28.57 -> 30.17 | Helped |
| optimize physical properties along with refined 3D Gaussians (ablation) | Vid2Sim · Synthetic dynamic reconstruction (12-object mean) | PSNR: 28.69 -> 30.17 | Helped |
| fine-tune LBS (ablation) | Vid2Sim · Synthetic dynamic reconstruction (12-object mean) | PSNR: 29.40 -> 30.17 | Helped |
| initialize physics with Stage I predictions instead of random values (ablation) | Vid2Sim · Synthetic dynamic reconstruction (12-object mean) | PSNR: 29.87 -> 30.17 | Helped |
Surprising findings
- Stage I physical predictions alone yield effective simulations even though Stage I appearance from 4-view LGM is weak (PSNR 19.77).
Limitations
- Reduced-order simulation limits modeling of complex materials such as fluids.
- Feed-forward model uses only a single front-view video and only elastic materials are predicted.
- Real-world evaluation uses only three captured sequences and relies on Spring-Gaus registration.
Experimental setup
Datasets: Objaverse Google Scanned Objects real-world captures (orange, bird, cup)
Benchmarks: Google Scanned Objects synthetic scenes (12 cases) real-world captures (orange, bird, cup)
Metrics: PSNR SSIM FoVVDP runtime
Baselines: PAC-NeRF Spring-Gaus GIC
Builds on: Simplicits 3D Gaussian Splatting PAC-NeRF Large Multi-view Gaussian Model VideoMAE
Closest papers by meaning
- MonoPhysics: Estimating Geometry, Appearance, and Physical Parameters from Monocular Videos (NeurIPS 2026 · similarity 0.88)
- DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation (NeurIPS 2024 · similarity 0.87)
- Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfels (NeurIPS 2024 · similarity 0.87)
- Physics-guided Shape-from-Template: Monocular Video Perception through Neural Surrogate Models (CVPR 2024 · similarity 0.87)
- Physically Compatible 3D Object Modeling from a Single Image (NeurIPS 2024 · similarity 0.86)
- OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.