Research map / 3D vision / Gaussian splatting
NeuMA: Neural Material Adaptor for Visual Grounding of Intrinsic Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeuMA, which learns a LoRA-style neural correction on top of pretrained neural constitutive laws inside an MPM simulator, trained end-to-end through a particle-bound 3D Gaussian Splatting renderer to ground intrinsic dynamics from visual observations.
Problem
Visual grounding of intrinsic material dynamics from videos: pure neural simulators can violate physics and generalize poorly, while expert-defined physical simulators may not match actual dynamics.
Approach
Model the material as a physics prior M0 (pretrained Neural Constitutive Law) plus a learned low-rank residual correction (LoRA), and bridge MPM particles to images via Particle-GS, a 3DGS variant whose Gaussian kernels are bound to multiple nearby simulation particles through a Mahalanobis-distance chi-squared binding.
Key results
- NeuMA achieves average Chamfer distance 1.31 (scaled by 1e4) vs 12.18 for NCLaw and 114.80 for PAC-NeRF across six synthetic benchmarks.
- NeuMA outperforms the particle-supervised variant (average 1.51) and the no-binding ablation (average 6.60) in grounding accuracy.
- On real-world data NeuMA's rendered sequences are reported as better aligned with observations than Spring-Gaus, both qualitatively and quantitatively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a residual material adaptor to the fixed physical prior | NeuMA · Six synthetic dynamics benchmarks, average | Chamfer distance: 12.18 -> 1.31 | Helped |
| Add a residual material adaptor rather than using the fixed prior alone (ablation) | NeuMA · BouncyBall | Chamfer distance: 1.87 -> 1.19 | Helped |
| Add a residual material adaptor rather than using the fixed prior alone (ablation) | NeuMA · JellyDuck | Chamfer distance: 3.63 -> 3.03 | Helped |
| Add a residual material adaptor rather than using the fixed prior alone (ablation) | NeuMA · RubberPawn | Chamfer distance: 1.42 -> 1.27 | Helped |
| Add a residual material adaptor rather than using the fixed prior alone (ablation) | NeuMA · ClayCat | Chamfer distance: 1.36 -> 1.00 | Helped |
| Add a residual material adaptor rather than using the fixed prior alone (ablation) | NeuMA · SandFish | Chamfer distance: 1.21 -> 0.65 | Helped |
| Add a residual material adaptor rather than using the fixed prior alone (ablation) | NeuMA · HoneyBottle | Chamfer distance: 1.23 -> 0.73 | Helped |
| Use particle binding instead of treating Gaussian kernels as physical particles (ablation) | NeuMA · JellyDuck | Chamfer distance: 28.42 -> 3.03 | Helped |
| Use particle binding instead of treating Gaussian kernels as physical particles (ablation) | NeuMA · BouncyBall | Chamfer distance: 3.34 -> 1.19 | Helped |
| Use particle binding instead of treating Gaussian kernels as physical particles (ablation) | NeuMA · RubberPawn | Chamfer distance: 4.62 -> 1.27 | Helped |
| Use particle binding instead of treating Gaussian kernels as physical particles (ablation) | NeuMA · ClayCat | Chamfer distance: 1.26 -> 1.00 | Helped |
| Use particle binding instead of treating Gaussian kernels as physical particles (ablation) | NeuMA · SandFish | Chamfer distance: 0.97 -> 0.65 | Helped |
Surprising findings
- NeuMA sometimes beats the variant trained with ground-truth 3D particle supervision (NeuMA w/ P.S.), which the authors attribute to the residual preserving the prior.
- Performance drops sharply when the physical prior is completely wrong (e.g., StVK + Drucker-Prager for an elastic object), with Chamfer distance rising to 30.26 on RubberPawn.
Limitations
- Requires multi-view surface reconstruction with calibrated cameras to obtain initial particles; struggles in complex scenes.
- Cumulative simulation errors degrade long-horizon future prediction.
- Motion blur in visual supervision degrades grounding performance.
Experimental setup
Datasets: Synthetic benchmarks BouncyBall, JellyDuck, RubberPawn, ClayCat, HoneyBottle, SandFish (MPM-simulated, Blender-rendered) Spring-Gaus real-world data (Bun, Burger, Dog, Pig)
Benchmarks: BouncyBall JellyDuck RubberPawn ClayCat SandFish HoneyBottle Spring-Gaus real-world scenes
Metrics: L2 Chamfer distance PSNR SSIM LPIPS
Baselines: PAC-NeRF NCLaw NCLaw + Particle-GS Spring-Gaus NeuMA w/o Bind NeuMA w/o delta M
Builds on: NCLaw PAC-NeRF PhysGaussian 3D Gaussian Splatting LoRA
Closest papers by meaning
- DEL: Discrete Element Learner for Learning 3D Particle Dynamics with Neural Rendering (NeurIPS 2024 · similarity 0.88)
- Learning 3D Particle-based Simulators from RGB-D Videos (ICLR 2024 · similarity 0.87)
- PBR-NeRF: Inverse Rendering with Physics-Based Neural Fields (CVPR 2025 · similarity 0.87)
- OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation (ICLR 2025 · similarity 0.86)
- DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors (AAAI 2025 · similarity 0.85)
- Latent Intuitive Physics: Learning to Transfer Hidden Physics from A 3D Video (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.