Research map / 3D vision / 3D generation
HoloScene: Simulation‑Ready Interactive 3D Worlds from a Single Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
HoloScene reconstructs simulation-ready, physically plausible interactive 3D scene graphs from a single video via energy-based optimization combining generative sampling, tree search, and Isaac Sim stability checks.
Problem
Digitizing indoor scenes from video into interactive digital twins that have complete geometry, object completeness, physical stability, and photorealistic rendering remains unsolved; existing methods lack one or more of these.
Approach
Represent a scene as a scene graph (instance neural SDFs, Gaussians-on-Mesh, physical params, support/beside/collide relations) and recover it by minimizing an energy combining observation, generative completion (Wonder3D after LaMa inpainting), geometry non-penetration, and simulator-based stability terms, using a three-stage pipeline of gradient optimization, sampling plus breadth-first tree sea
Key results
- On Replica, HoloScene reaches 95.7% ground-stable and 81.7% all-stable instances versus 39.4% for the best baseline (ObjSDF++, 78.3%/39.4%) and 100% object reconstruction ratio.
- On iGibson scenes, HoloScene achieves 71.1% all-stable with PSNR 25.88, versus 36.1% for ObjSDF++ and 5.2% for PhyRecon.
- Object-level iGibson evaluation gives CD 3.17 and F1 81.31, better than ObjSDF++ (3.52, 79.03), PhyRecon and DP-Recon on occluded geometry.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HoloScene instead of DP-Recon | Scene reconstruction · Replica | Stable (All)%: 39.4 -> 81.7 | Helped |
| Use HoloScene instead of DP-Recon | Scene reconstruction · ScanNet++ | Stable (All)%: 28.2 -> 70.6 | Helped |
| Use HoloScene instead of PhyRecon | Scene reconstruction · iGibson scene | Stable (All)%: 5.2 -> 71.1 | Helped |
| Use HoloScene instead of DP-Recon | Scene reconstruction · Replica | CD: 3.45 -> 4.05 | Hurt |
| Use HoloScene instead of DP-Recon | Scene reconstruction · ScanNet++ | PSNR: 21.44 -> 25.88 | Hurt |
| Replace textured mesh with GoM (ablation) | HoloScene without physics energy or scene graph · Replica | PSNR: 26.45 -> 28.30 | Helped |
| Add physics energy (ablation) | GoM without scene graph relationships · Replica | Stable (All)%: 43.7 -> 69.0 | Helped |
| Add scene graph inter-object relationships (ablation) | GoM with physics energy · Replica | Stable (All)%: 69.0 -> 81.7 | Helped |
| Add physics energy and scene graph relationships (ablation) | GoM without physics energy or scene graph · iGibson objects | Stable (All)%: 41.2 -> 71.1 | Helped |
| Optimize for physical plausibility with physics energy and scene graph (ablation) | HoloScene · Replica | CD: 3.50 -> 4.05 | Hurt |
Surprising findings
- Optimizing for physical plausibility trades off some scene-level pixel alignment; DP-Recon achieves better scene-level CD and F1 on Replica than HoloScene.
- ObjectSDF++ has higher ground-stable ratio on ScanNet++ (28.2%) than the reported stable ratio of... and object-level stability gains depend strongly on the physics energy and scene graph.
Limitations
- Handles only static indoor scenes; dynamic scenes and large outdoor environments remain challenging.
- Physical plausibility optimization can reduce pixel-level alignment with observations.
Experimental setup
Datasets: Replica ScanNet++ iGibson
Benchmarks: Replica ScanNet++ iGibson
Metrics: Chamfer Distance F-Score Normal Consistency PSNR SSIM LPIPS object reconstruction ratio stable object ratio
Baselines: ObjectSDF++ PhyRecon DP-Recon CAST
Builds on: ObjectSDF++ PhyRecon DP-Recon Gaussian splatting Wonder3D
Closest papers by meaning
- PhysGen3D: Crafting a Miniature Interactive World from a Single Image (CVPR 2025 · similarity 0.87)
- DRAWER: Digital Reconstruction and Articulation With Environment Realism (CVPR 2025 · similarity 0.86)
- LiteReality: Graphic-Ready 3D Scene Reconstruction from RGB-D Scans (NeurIPS 2025 · similarity 0.86)
- Guiding Human-Object Interactions with Rich Geometry and Relations (CVPR 2025 · similarity 0.86)
- Video2Game: Real-time Interactive Realistic and Browser-Compatible Environment from a Single Video (CVPR 2024 · similarity 0.85)
- Simulation-Ready Compositional 3D Scene Reconstruction from a Single Image (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.