Research map / 3D vision / NeRF and radiance fields
ReconFusion: 3D Reconstruction with Diffusion Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReconFusion, which regularizes a Zip-NeRF reconstruction with a multiview-conditioned latent diffusion prior (PixelNeRF + CLIP conditioning) to reconstruct real scenes from only a few photos.
Problem
NeRF reconstruction of real-world scenes requires tens to hundreds of images; with few input views, optimized NeRFs produce floaters, fog, and catastrophic failures at novel viewpoints.
Approach
Fine-tune a latent diffusion model for novel view synthesis conditioned on multiple posed input images (CLIP embeddings via cross-attention plus PixelNeRF feature maps concatenated to the U-Net input), then use it as a per-iteration multistep sampling regularizer (sample loss on rendered novel views) during NeRF optimization.
Key results
- With 3 input views on RealEstate10K, ReconFusion reaches 25.84 PSNR versus 23.89 for SimpleNeRF (best baseline).
- On CO3D with 3 views, ReconFusion achieves 19.59 PSNR versus 17.13 for ZeroNVS*.
- On mip-NeRF 360 with 9 views, ReconFusion reaches 18.19 PSNR versus 15.99 for ZeroNVS*; the paper reports outperforming all baselines across all datasets and view counts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ReconFusion instead of the strongest baseline, SimpleNeRF | few-view NeRF reconstruction · RealEstate10K 3-view | PSNR: 23.89 -> 25.84 | Helped |
| use ReconFusion instead of the strongest baseline, DiffusioNeRF | few-view NeRF reconstruction · LLFF 3-view | PSNR: 20.13 -> 21.34 | Helped |
| use ReconFusion instead of the strongest baseline, FreeNeRF | few-view NeRF reconstruction · DTU 9-view | PSNR: 25.56 -> 24.62 | Hurt |
| use ReconFusion instead of the strongest baseline, ZeroNVS* | few-view NeRF reconstruction · CO3D 3-view | PSNR: 17.13 -> 19.59 | Helped |
| use ReconFusion instead of the strongest baseline, ZeroNVS* | few-view NeRF reconstruction · mip-NeRF 360 3-view | PSNR: 14.44 -> 15.50 | Helped |
| replace PixelNeRF conditioning with pose conditioning (ablation) | ReconFusion diffusion model · In-domain datasets | PSNR: 25.15 -> 20.57 | Hurt |
| replace PixelNeRF conditioning with pose conditioning (ablation) | ReconFusion diffusion model · Out-domain datasets | PSNR: 19.82 -> 17.28 | Hurt |
| remove pretrained diffusion weights (ablation) | ReconFusion diffusion model with PixelNeRF conditioning · In-domain datasets | PSNR: 25.34 -> 25.15 | Hurt |
| remove pretrained diffusion weights (ablation) | ReconFusion diffusion model with PixelNeRF conditioning · Out-domain datasets | PSNR: 20.23 -> 19.82 | Hurt |
| use the standard SDS diffusion loss instead of multistep sampling (ablation) | ReconFusion reconstruction · 3D reconstruction | visual artifacts | Hurt |
| anneal tmin during multistep diffusion sampling (ablation) | ReconFusion reconstruction · 3D reconstruction | detail | Helped |
Surprising findings
- Standard score distillation sampling produced more artifacts than the proposed multistep diffusion loss.
- Training the diffusion model from scratch (without pretrained text-to-image weights) degraded results, and the ZeroNVS-style pose conditioning underperformed PixelNeRF feature-map conditioning.
Limitations
- Heavyweight diffusion model slows reconstruction significantly.
- Limited 3D outpainting ability compared with 2D hallucination capability.
- Balancing reconstruction and sample losses is tedious.
Experimental setup
Datasets: Objaverse CO3D MVImgNet RealEstate10K LLFF DTU mip-NeRF 360
Benchmarks: RealEstate10K LLFF DTU CO3D mip-NeRF 360
Metrics: PSNR SSIM LPIPS
Baselines: Zip-NeRF DiffusioNeRF FreeNeRF SimpleNeRF ZeroNVS SparseFusion RegNeRF
Builds on: Zip-NeRF Latent Diffusion Models Zero-1-to-3 SparseFusion PixelNeRF
Closest papers by meaning
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.90)
- Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.88)
- Decoupling Scattering: Pseudo-Label Guided NeRF for Scenes with Scattering Media (AAAI 2025 · similarity 0.88)
- DIFIX3D+: Improving 3D Reconstructions with Single-Step Diffusion Models (CVPR 2025 · similarity 0.88)
- PBR-NeRF: Inverse Rendering with Physics-Based Neural Fields (CVPR 2025 · similarity 0.88)
- Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.