Research map / 3D vision / 3D generation
O^2-Recon: Completing 3D Reconstruction of Occluded Objects in the Scene with a Pre-trained 2D Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reconstructs complete object surfaces from occluded RGB-D scene videos by inpainting occluded object regions with Stable Diffusion, using human-sketched masks, and optimizing a cascaded neural implicit SDF with CLIP semantic loss.
Problem
Object-level 3D reconstruction from scene-level RGB-D videos is incomplete because objects are occluded and camera coverage is limited, leaving unseen surfaces missing or degenerate.
Approach
Use a pre-trained 2D diffusion inpainting model to fill occluded object regions in frames, with masks produced by a light human-in-the-loop sketching and depth-based re-projection; reconstruct each object with a NeuS-style implicit surface using a coarse-to-fine cascaded SDF network and a CLIP-based semantic consistency loss on novel views.
Key results
- On ScanNet GT, O2-Recon reaches F-score 0.715 and Chamfer distance 4.45 cm, versus 0.636 and 10.40 for vMap*.
- On Scan2CAD GT, O2-Recon reaches F-score 0.568 and Chamfer distance 6.15 cm, versus 0.471 and 13.23 for vMap*.
- Ablations show color inpainting improves Scan2CAD Chamfer distance from 9.73 (projected masks only) to 6.80, and cascaded SDF plus semantic loss reduce it further to 6.15.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add projected masks to V1 (ablation) | O²-Recon · ScanNet | F-score: 0.718 -> 0.697 | Hurt |
| Add projected masks to V1 (ablation) | O²-Recon · Scan2CAD | F-score: 0.502 -> 0.511 | Helped |
| Add color in-painting to V2 (ablation) | O²-Recon · ScanNet | F-score: 0.697 -> 0.720 | Helped |
| Add color in-painting to V2 (ablation) | O²-Recon · Scan2CAD | Chamfer distance: 8.72 -> 6.80 | Helped |
| Add cascaded SDF to V3 (ablation) | O²-Recon · ScanNet | F-score: 0.720 -> 0.713 | Hurt |
| Add cascaded SDF to V3 (ablation) | O²-Recon · Scan2CAD | F-score: 0.553 -> 0.560 | Helped |
| Add semantic loss to V4 (ablation) | O²-Recon · ScanNet | Chamfer distance: 4.53 -> 4.45 | Helped |
| Add semantic loss to V4 (ablation) | O²-Recon · Scan2CAD | F-score: 0.560 -> 0.568 | Helped |
| Add O²-Recon instead of vMap* | object reconstruction · ScanNet GT | F-score: 0.636 -> 0.715 | Helped |
| Add O²-Recon instead of vMap* | object reconstruction · Scan2CAD GT | Chamfer distance: 13.23 -> 6.15 | Helped |
Surprising findings
- Plugging SDFusion 3D completion into MonoSDF degrades its performance (ScanNet F-score 0.627 to 0.509).
Limitations
- Requires 1-3 user-sketched masks per object, so some human engagement is needed.
- Per-object optimization takes about 4 hours per object.
- Depth and projected masks from inpainted depth can be inaccurate, needing morphological post-processing.
Experimental setup
Datasets: ScanNet Scan2CAD
Benchmarks: ScanNet Scan2CAD
Metrics: F-score accuracy distance completeness distance Chamfer distance
Baselines: MonoSDF FroDO Scan2CAD vMap SDFusion
Builds on: NeuS MonoSDF vMap NeuRIS
Closest papers by meaning
- In-Hand 3D Object Reconstruction from a Monocular RGB Video (AAAI 2024 · similarity 0.88)
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.88)
- Coherent 3D Scene Diffusion From a Single RGB Image (NeurIPS 2024 · similarity 0.88)
- UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections (NeurIPS 2024 · similarity 0.87)
- OFER: Occluded Face Expression Reconstruction (CVPR 2025 · similarity 0.87)
- Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.