Research map / 3D vision / 3D generation

Simulation-Ready Compositional 3D Scene Reconstruction from a Single Image

NeurIPS 2026 · 3D vision › 3D generation · approach: Learning physics from video

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Puts a MuJoCo physics simulation in the loop of single-image compositional 3D scene reconstruction, using gravity-induced displacement to drive gravity-axis stretching or LoRA-DPO-finetuned SAM3D amodal resampling, yielding simulation-stable scenes.

Problem

Single-image multi-object 3D reconstruction produces per-object shapes and poses that interpenetrate, float, or sink when placed in a physics simulator, and existing physics-aware methods only correct layout post hoc without fixing geometry.

Approach

Treats the physics engine as a diagnostic probe: gravity-axis displacement normalized by OBB height (rho) decides between a lightweight gravity-axis stretch for small errors and OBB-guided amodal shape resampling (SAM3D with physics-informed crop, LoRA fine-tuned via flow-matching DPO) for large errors, applied sequentially per object.

Key results

• On GenWild, Ours reaches ABO_fh/fo 0.507 and penetration ratio 2.5% versus 0.218 and 16.2% for SAM3D.
• On AriaDigitalTwin 3D ABO_fh/fo, Ours scores 0.293 versus 0.190 for the best baseline with our background (Gen3DSR).
• Amodal resampling achieves substantially higher VLM-judged Elo and win rate than raw SAM3D and simple stretching (table reference only; values not given in text).

Surprising findings

• Standard ABO rewards physically interpenetrating objects, so the authors introduce stricter ABO_fh/fo variants that zero out invalid instances.

Limitations

• Sequential protocol cannot revise early object estimates using evidence from later objects.
• Relies on physics-simulation signals and a rho threshold of 0.15 chosen heuristically.
• Pipeline depends on many foundation components (VLM tags, SAM3, MoGe, FoundationPose), so errors can compound.

Experimental setup

Datasets: GraspClutter6D Aria Digital Twin GenWild

Benchmarks: GraspClutter6D AriaDigitalTwin GenWild

Metrics: ABO ABO_fh/fo mean displacement peak energy penetration ratio IoU stability ratio 3D ABO

Baselines: SAM3D Gen3DSR 3D-RE-GEN

Builds on: SAM3D Diffusion-DPO DSO FoundationPose MoGe

Closest papers by meaning

• PRISM: Programming Interactive Scenes from Monocular Images for Embodied Simulation (NeurIPS 2026 · similarity 0.88)
• DecoupledGaussian: Object-Scene Decoupling for Physics-Based Interaction (CVPR 2025 · similarity 0.87)
• Physically Compatible 3D Object Modeling from a Single Image (NeurIPS 2024 · similarity 0.86)
• URDF-Anything+: End-to-End Generation for Simulation-Ready Articulated Assets (NeurIPS 2026 · similarity 0.86)
• MonoPhysics: Estimating Geometry, Appearance, and Physical Parameters from Monocular Videos (NeurIPS 2026 · similarity 0.85)
• PhysX-3D: Physical-Grounded 3D Asset Generation (NeurIPS 2025 · similarity 0.85)