Research map / 3D vision / Face and shape
Learning rigid-body simulators over implicit shapes for large-scale scenes and vision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SDF-Sim, a GNN-based rigid-body simulator that uses per-object learned MLP signed-distance functions and object-to-node SDF collision edges to scale learned simulation to hundreds of objects and up to 1.1M nodes.
Problem
Learned GNN rigid-body simulators rely on mesh collision edges whose count grows quadratically with nodes, so they run out of memory on scenes with many objects or detailed meshes.
Approach
Represent each object by a learned MLP SDF and connect each surface node only to the opposing object's centre node via SDF-queried collision edges, making inter-object edges linear in the number of nodes instead of quadratic, and allowing SDFs to be extracted from multi-view images via VolSDF.
Key results
- On Movi-B SDF-Sim has 201 collision edges and 38.8 MiB peak memory versus 1409.87 edges and 63.83 MiB for FIGNet*, with 0.16 m translation RMSE versus 0.13 m for FIGNet*.
- SDF-Sim simulates 512 spheres on a single V100 GPU, where FIGNet and FIGNet* run out of memory beyond 140 and 160 objects.
- SDF-Sim runs simulations with 851k, 384k and 1.1M nodes where no other learned baseline can run.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SDF-based object-to-node collision edges instead of mesh-based collision edges | SDF-Sim · Movi-B | Runtime per step (ms): 0.22 $\pm$ 0.0077 -> \textbf{0.14} $\pm$ 0.004 | Helped |
| use SDF-based object-to-node collision edges instead of mesh-based collision edges | SDF-Sim · Movi-C | Runtime per step (ms): 0.30 $\pm$ 0.0163 -> \textbf{0.17} $\pm$ 0.0038 | Helped |
| use SDF-based object-to-node collision edges instead of mesh-based collision edges | SDF-Sim · Movi-B | Translation RMSE (m): \textbf{0.13} $\pm$ 0.008 -> 0.16 $\pm$ 0.0124 | Hurt |
| use SDF-based object-to-node collision edges instead of mesh-based collision edges | SDF-Sim · Movi-C | Translation RMSE (m): \textbf{0.18} $\pm$ 0.0097 -> 0.24 $\pm$ 0.0151 | Hurt |
| increase SDF MLP width from 32 to 64 or 128 units per layer (ablation) | SDF-Sim trained on Movi-B · Movi-B | Translation and rotation error | No clear effect |
| increase SDF MLP width from 32 to 64 or 128 units per layer (ablation) | SDF-Sim trained on Movi-B · Movi-B SDF shape estimates | SDF estimation error | Helped |
| replace collision-mesh nodes with surface nodes sampled from the SDF (K=7^3) (ablation) | SDF-Sim · Movi-C | Translation RMSE (m): 0.24 -> 0.23 | No clear effect |
| replace collision-mesh nodes with surface nodes sampled from the SDF (K=7^3) (ablation) | SDF-Sim · Movi-C | nodes per scene: 3530.13 -> 934.19 | Helped |
| replace learned signed distances with unsigned distances computed from the mesh via BVH (ablation) | SDF-Sim · Movi-C | Translation RMSE (m): 0.23 with mesh-based UDFs versus 0.24 with learned SDF | No clear effect |
| add Eikonal regularization to SDF training (ablation) | SDF-Sim SDF model · SDF shape reconstruction | shape quality | Hurt |
| add periodic encodings to the SDF MLP inputs (ablation) | SDF-Sim SDF model · SDF shape reconstruction | shape artifacts | Hurt |
Surprising findings
- SDF-Sim generalizes better than FIGNet* to large scenes despite being trained only on 10-object scenes and having slightly higher error on small benchmarks.
- Using unsigned distances from BVH gives similar translation RMSE (0.23 m) to learned signed SDFs (0.24 m) on Movi-C.
Limitations
- Requires training a separate SDF MLP for every new shape (about 18 hours per Movi-C mesh on a V100).
- Slightly higher translation and rotation error than FIGNet/FIGNet* on small Movi-B/C benchmarks.
- Only rigid-rigid interactions are studied.
Experimental setup
Datasets: Kubric Movi-B Kubric Movi-C Spheres-in-Bowl (custom, PyBullet ground truth) Garden scene (VolSDF-distilled SDF)
Benchmarks: Kubric Movi-B Kubric Movi-C Spheres-in-Bowl
Metrics: translation RMSE rotation error peak memory runtime per step number of collision edges penetration distance rollout RMSE
Baselines: FIGNet FIGNet* DPI MeshGraphNets MGN-LargeRadius
Builds on: MeshGraphNets FIGNet DeepMind learned simulators (graph network simulator) VolSDF Kubric
Closest papers by meaning
- Point Cloud Sequence Encoding for Material-conditioned Graph Network Simulators (NeurIPS 2026 · similarity 0.85)
- EGODE: An Event-attended Graph ODE Framework for Modeling Rigid Dynamics (NeurIPS 2024 · similarity 0.85)
- Degrees of Freedom Matter: Inferring Dynamics from Point Trajectories (CVPR 2024 · similarity 0.84)
- URDF-Anything+: End-to-End Generation for Simulation-Ready Articulated Assets (NeurIPS 2026 · similarity 0.84)
- Learning 3D Particle-based Simulators from RGB-D Videos (ICLR 2024 · similarity 0.84)
- Object Dynamics Modeling with Hierarchical Point Cloud-based Representations (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.