Research map / 3D vision / 3D generation
Decompositional Neural Scene Reconstruction with Generative Diffusion Prior
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DP-RECON, which uses Stable Diffusion SDS priors with a learned visibility grid to reweight SDS loss, improving sparse-view decompositional scene reconstruction of objects and background.
Problem
Decompositional neural scene reconstruction from sparse views fails to recover complete geometry and texture in underconstrained and occluded regions.
Approach
Applies per-object score distillation sampling from a pretrained 2D diffusion model to supervise novel views during object-compositional NeRF/SDF reconstruction, and uses a learnable visibility grid derived from accumulated transmittance to dynamically reweight per-pixel SDS loss so generative guidance does not conflict with observed input views.
Key results
- On Replica with 10 views, object reconstruction CD improves from 7.49 (ObjectSDF++) to 5.54, and mIoU rises to 88.21.
- With 10 input views, the method's object reconstruction beats baselines run with 100 views on heavily occluded scenes (Fig. 1).
- Visibility-guided SDS plus appearance prior raises ScanNet++ scene-level CD from 10.18 (Oursgeo) to a comparable setting and improves MUSIQ from 38.43 to 45.71.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add geometry diffusion prior (ablation) | DP-RECON · Replica | object reconstruction mIoU: 73.31 -> 84.45 | Helped |
| add visibility guidance for geometry prior (ablation) | DP-RECON with GP · Replica | object reconstruction mIoU: 84.45 -> 88.21 | Helped |
| add appearance diffusion prior without visibility guidance (ablation) | DP-RECON with GP · Replica | PSNR: 24.83 -> 22.83 | Hurt |
| add visibility guidance for appearance prior (ablation) | DP-RECON with GP, VG, and AP · Replica | PSNR: 23.42 -> 24.52 | Helped |
| use DP-RECON instead of ObjectSDF++ | Replica · Replica | CD: 8.57 -> 7.91 | Helped |
| use DP-RECON instead of ObjectSDF++ | Replica · Replica | MUSIQ: 41.42 -> 49.22 | Helped |
| use DP-RECON instead of ObjectSDF++ | ScanNet++ · ScanNet++ | CD: 10.67 -> 10.18 | Helped |
| use DP-RECON instead of ObjectSDF++ | ScanNet++ · ScanNet++ | MUSIQ: 33.64 -> 45.71 | Helped |
Surprising findings
- The visibility-guided weighting slightly lowers PSNR on Replica (24.52 vs 25.08 for geometry-only) while raising no-reference MUSIQ, indicating a trade-off between faithfulness and generative detail.
Limitations
- Relies on SDS optimization, which is known for oversaturation, oversmoothing and instability (noted by the authors).
- Evaluated on only 8 Replica and 6 ScanNet++ scenes with per-scene optimization.
Experimental setup
Datasets: Replica ScanNet++
Benchmarks: Replica ScanNet++
Metrics: Chamfer Distance F-Score Normal Consistency mIoU PSNR SSIM LPIPS MUSIQ
Baselines: ZeroNVS FreeNeRF MonoSDF RICO ObjectSDF++
Builds on: DreamFusion MonoSDF ObjectSDF++ VolSDF ReconFusion
Closest papers by meaning
- Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.91)
- Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views (AAAI 2024 · similarity 0.90)
- ReconFusion: 3D Reconstruction with Diffusion Priors (CVPR 2024 · similarity 0.90)
- Coherent 3D Scene Diffusion From a Single RGB Image (NeurIPS 2024 · similarity 0.90)
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.90)
- In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.