Research map / 3D vision / 3D generation
Zero-Shot Scene Reconstruction from Single Images with Deep Prior Assembly
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Deep Prior Assembly reconstructs 3D scenes from a single image zero-shot by chaining frozen large models (Grounded-SAM, Stable Diffusion, Open-CLIP, Shap-E, Omnidata) with an optimization-based 7-DoF pose/scale registration.
Problem
Single-image 3D scene reconstruction is hard because learned end-to-end methods trained on limited image-scene pairs fail to generalize to open-world images.
Approach
Decompose the task into sub-tasks, assign each to a frozen off-the-shelf large model (detection/segmentation, enhancement/inpainting, 2D-to-3D generation, depth), and recover layout by optimizing 7-DoF poses/scales of generated shapes using both 3D depth Chamfer loss and 2D mask Chamfer loss, with a RANSAC-like restart scheme.
Key results
- On 3D-Front, achieves CDL1-S 0.109 and F-Score 35.67 versus PanoRecon's 0.120 and 31.94 (PanoRecon has the best CDL1 at 0.125 vs. 0.134 for ours).
- On BlendSwap, F-Score 73.19 vs. 26.93 for Total3D; on Replica, F-Score 70.48 vs. 25.42 for Mesh R-CNN.
- Ablation on Replica: removing 3D matching raises CDL1-S from 0.113 to 0.199 and drops F-Score from 70.48 to 56.08.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use deep prior assembly instead of PanoRecon | 3D-Front scene reconstruction · 3D-Front | CDL1-S: 0.120 -> 0.109 | Helped |
| use deep prior assembly instead of PanoRecon | 3D-Front scene reconstruction · 3D-Front | CDL1: 0.125 -> 0.134 | Hurt |
| use deep prior assembly instead of PanoRecon | 3D-Front scene reconstruction · 3D-Front | F-Score: 31.94 -> 35.67 | Helped |
| use deep prior assembly instead of Total3D | BlendSwap scene reconstruction · BlendSwap | CDL1-S: 0.133 -> 0.106 | Helped |
| use deep prior assembly instead of Total3D | BlendSwap scene reconstruction · BlendSwap | CDL1: 0.400 -> 0.089 | Helped |
| use deep prior assembly instead of Total3D | BlendSwap scene reconstruction · BlendSwap | F-Score: 26.93 -> 73.19 | Helped |
| use deep prior assembly instead of Mesh R-CNN | Replica scene reconstruction · Replica | CDL1-S: 0.268 -> 0.113 | Helped |
| use deep prior assembly instead of Mesh R-CNN | Replica scene reconstruction · Replica | CDL1: 0.408 -> 0.110 | Helped |
| use deep prior assembly instead of Mesh R-CNN | Replica scene reconstruction · Replica | F-Score: 25.42 -> 70.48 | Helped |
| add Stable-Diffusion enhancement and inpainting (ablation) | Deep prior assembly · Replica | F-Score: 67.22 -> 70.48 | Helped |
| add 2D matching constraint (ablation) | Deep prior assembly · Replica | F-Score: 68.42 -> 70.48 | Helped |
| add 3D matching constraint (ablation) | Deep prior assembly · Replica | F-Score: 56.08 -> 70.48 | Helped |
Surprising findings
- PanoRecon, trained on 3D-Front, degrades substantially on open-world BlendSwap and Replica images (F-Score 17.11 on BlendSwap), indicating poor out-of-distribution generalization of supervised methods.
Limitations
- Reconstructed 3D instances may not align in color with the input 2D instances due to randomness in diffusion-based enhancement.
- Scale and shift of Omnidata relative depth can cause distortion in reconstructions; metric depth estimators suggested as remedy.
- Relies on the use of ground-truth depth to solve scale and shift for each dataset.
Experimental setup
Datasets: 3D-Front Replica BlendSwap ScanNet
Benchmarks: 3D-Front Replica BlendSwap ScanNet
Metrics: Chamfer Distance (CDL1) single-direction Chamfer Distance (CDL1-S) F-Score
Baselines: Mesh R-CNN Total3D PanoRecon BUOL Uni-3D ScenePrior One-2-3-45
Builds on: Grounded-SAM Stable Diffusion Shap-E Omnidata Open-CLIP
Closest papers by meaning
- 3D Reconstruction with Generalizable Neural Fields using Scene Priors (ICLR 2024 · similarity 0.88)
- Zero-shot Depth Completion via Test-time Alignment with Affine-invariant Depth Prior (AAAI 2025 · similarity 0.87)
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (CVPR 2024 · similarity 0.87)
- LEAP: Liberate Sparse-View 3D Modeling from Camera Poses (ICLR 2024 · similarity 0.86)
- MP-SfM: Monocular Surface Priors for Robust Structure-from-Motion (CVPR 2025 · similarity 0.86)
- ECoDepth: Effective Conditioning of Diffusion Models for Monocular Depth Estimation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.