Research map / 3D vision / 3D generation
Learning 3D Particle-based Simulators from RGB-D Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns a 3D latent particle-based simulator end to end from posed multi-view RGB-D videos, using a hierarchical GNN dynamics model and an annular-kernel NeRF-style renderer, with no state or mask supervision.
Problem
Learning physical simulators directly from visual observations without privileged state information (object poses, geometry, masks), while retaining 3D structure, editability, and long-horizon prediction.
Approach
Jointly learn an encoder that unprojects RGB-D pixels into a latent 3D particle set, a hierarchical GNN with abstract nodes that predicts particle deltas without correspondences, and a volumetric renderer that decodes particles into images, all trained end to end on a multi-step pixel loss.
Key results
- VPD achieves the best average PSNR on MuJoCo Block (33.076), Deformable Block (31.520), Multi (31.221), and Collision (28.725), and on MOVi-C (23.142) over 32-step rollouts.
- With only 16 training trajectories on MuJoCo Block, VPD reaches PSNR 31.22, about 1.85 points below its 256-trajectory result.
- VPD trained with a single RGB-D camera performs comparably to 9 cameras on MuJoCo Block.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a 3D particle representation and GNN dynamics instead of a global latent vector and MLP (ablation) | VPD · MuJoCo Block, Deformable Block, Deformable Multi, Deformable Collision, Kubric MOVi-A/B/C | PSNR: 29.601 -> 33.076; 26.969 -> 31.520; 26.636 -> 31.221; 22.394 -> 28.725; 27.604 -> 29.014; 26.201 -> 27.194; 20.068 -> 23 | Helped |
| Train the encoder/decoder end to end with dynamics instead of sequentially pretraining and freezing them (ablation) | VPD · MuJoCo Block, Deformable Block, Deformable Multi, Deformable Collision, Kubric MOVi-A/B/C | PSNR: 25.858 -> 33.076; 30.854 -> 31.520; 31.256 -> 31.221; 27.911 -> 28.725; 28.613 -> 29.014; 25.921 -> 27.194; 21.854 -> 23 | Mixed |
| Train with 16 trajectories instead of 256 (ablation) | VPD · MuJoCo Block | PSNR: 31.22; 1.85 points less | Hurt |
| Use one RGB-D camera instead of 9 (ablation) | VPD · MuJoCo Block | PSNR | No clear effect |
| Add an additional feature encoder to process nearby particle features (ablation) | VPD · MuJoCo Block | PSNR | No clear effect |
| Normalize feature contributions by relative distances to nearby particles (ablation) | VPD · MuJoCo Block | PSNR | No clear effect |
Surprising findings
- SlotFormer outperforms VPD on MOVi-A and MOVi-B in PSNR (30.934 and 28.810 vs 29.014 and 27.194 for VPD).
- Even with a single RGB-D camera, VPD performs comparably to 9 cameras, which the authors say they cannot fully explain.
Limitations
- Blurring over very long rollouts remains a challenge.
- Requires RGB-D video with depth supervision; predicted-depth experiment is only preliminary.
- Baselines (NeRF-dy) could not be run directly since code and data were unavailable, so ablations were used instead.
Experimental setup
Datasets: MuJoCo block Kubric MOVi-A Kubric MOVi-B Kubric MOVi-C Deformables (Block, Multi, Collision)
Benchmarks: MuJoCo Block Kubric MOVi-A Kubric MOVi-B Kubric MOVi-C Deformable Block Deformable Multi Deformable Collision
Metrics: PSNR SSIM
Baselines: SlotFormer NeRF-dy (Global ablation) Sequential ablation
Builds on: NeRF NeRF-dy PointNeRF SAVi SlotFormer Encode-Process-Decode
Closest papers by meaning
- DEL: Discrete Element Learner for Learning 3D Particle Dynamics with Neural Rendering (NeurIPS 2024 · similarity 0.90)
- Latent Intuitive Physics: Learning to Transfer Hidden Physics from A 3D Video (ICLR 2024 · similarity 0.89)
- NeuMA: Neural Material Adaptor for Visual Grounding of Intrinsic Dynamics (NeurIPS 2024 · similarity 0.87)
- DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors (AAAI 2025 · similarity 0.87)
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.86)
- FreeGave: 3D Physics Learning from Dynamic Videos by Gaussian Velocity (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.