Research map / 3D vision / 3D generation
DRAWER: Digital Reconstruction and Articulation With Environment Realism
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Converts a video of a static indoor scene into a photorealistic, articulated, interactive digital twin via a dual SDF + Gaussian splat representation and an articulation module, demonstrated in Unreal Engine and real-to-sim-to-real robot learning.
Problem
Automatically creating interactable, photorealistic and geometrically precise digital twins of real indoor scenes from passive video; existing methods trade off visual fidelity against geometry or interactivity.
Approach
A dual scene representation that anchors 3D Gaussian splats to the zero level-set of a neural SDF (mesh faces) with straight-through clipping, combined with an articulation module that estimates joint types/axes via foundation models and fits shape-primitive templates to hidden parts, enabling simulatable objects.
Key results
- Novel view synthesis PSNR 27.80 / SSIM 0.912 / LPIPS 0.159, versus Video2Game (Mesh) 22.63 and BakedSDF 21.11, while 3DGS reaches higher PSNR (30.42) but lacks interactivity.
- Articulation understanding precision 97.2% and recall 84.0% versus URDFormer 93.5% / 46.4%.
- Articulated motion EMD 1.41e-5 versus 17.7e-5 for KlingAI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DRAWER's dual scene representation | DRAWER · Novel view synthesis | PSNR: 27.80 vs. 30.42 | Mixed |
| use DRAWER's dual scene representation | DRAWER · Novel view synthesis | SSIM: 0.912 vs. 0.954 | Mixed |
| use DRAWER's dual scene representation | DRAWER · Novel view synthesis | LPIPS: 0.159 vs. 0.126 | Mixed |
| add Gaussian splatting to a textured mesh (ablation) | DRAWER · Novel view synthesis | PSNR: 19.71 -> 25.87 | Helped |
| add Gaussian splatting to a textured mesh (ablation) | DRAWER · Novel view synthesis | SSIM: 0.785 -> 0.889 | Helped |
| add Gaussian splatting to a textured mesh (ablation) | DRAWER · Novel view synthesis | LPIPS: 0.323 -> 0.276 | Helped |
| allow Gaussian splats to move (ablation) | DRAWER · Novel view synthesis | PSNR: 25.87 -> 26.49 | Helped |
| allow Gaussian splats to move (ablation) | DRAWER · Novel view synthesis | SSIM: 0.889 -> 0.896 | Helped |
| allow Gaussian splats to move (ablation) | DRAWER · Novel view synthesis | LPIPS: 0.276 -> 0.193 | Helped |
| use a straight-through estimator (ablation) | DRAWER · Novel view synthesis | PSNR: 26.49 -> 27.80 | Helped |
| use a straight-through estimator (ablation) | DRAWER · Novel view synthesis | SSIM: 0.896 -> 0.912 | Helped |
| use a straight-through estimator (ablation) | DRAWER · Novel view synthesis | LPIPS: 0.193 -> 0.159 | Helped |
Surprising findings
- 3DGS achieves the best pure rendering quality (PSNR 30.42) but yields scenes that are neither realistic nor interactive due to poor geometry.
Limitations
- Relies on the reconstruction and joint estimation of foundation models (GPT-4o, 3DOI) whose errors propagate.
- Relightable environments are not handled, as noted in the conclusion.
- Evaluated on only six kitchen scenes.
Experimental setup
Datasets: six self-captured kitchen videos
Benchmarks: six self-captured kitchen scenes
Metrics: PSNR SSIM LPIPS precision recall EA-Score EMD trajectory error
Baselines: Nerfacto Video2Game BakedSDF 3DGS 2DGS URDFormer Digital Cousin 3DOI KlingAI
Builds on: 3D Gaussian splatting neural SDF (VolSDF/NeuS-style surface rendering) BakedSDF Video2Game 3DOI
Closest papers by meaning
- LiteReality: Graphic-Ready 3D Scene Reconstruction from RGB-D Scans (NeurIPS 2025 · similarity 0.88)
- Video2Game: Real-time Interactive Realistic and Browser-Compatible Environment from a Single Video (CVPR 2024 · similarity 0.88)
- HoloScene: Simulation‑Ready Interactive 3D Worlds from a Single Video (NeurIPS 2025 · similarity 0.86)
- Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model (ICLR 2025 · similarity 0.86)
- Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting (NeurIPS 2024 · similarity 0.85)
- PRISM: Programming Interactive Scenes from Monocular Images for Embodied Simulation (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.