Research map / 3D vision / NeRF and radiance fields
ExtraNeRF: Visibility-Aware View Extrapolation of Neural Radiance Fields with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extrapolates NeRF views from six or fewer inputs by using visibility-masked, per-scene fine-tuned latent diffusion inpainting plus a second diffusion enhancement pass to sharpen the NeRF.
Problem
View extrapolation with NeRF from a few narrowly spaced input views, where disoccluded and unobserved regions are blurry or missing.
Approach
Track visibility via NeRF transmittance to find unobserved regions, inpaint them in virtual views with a per-scene fine-tuned latent diffusion inpainter, then run a second fine-tuned diffusion enhancer to sharpen and color-correct the inpainted NeRF.
Key results
- On LLFF view extrapolation, ExtraNeRF reaches LPIPS 0.269 and KID 0.0154 versus 0.284 and 0.0156 for the best baseline SPIn-NeRF.
- On Tanks & Temples, ExtraNeRF achieves LPIPS 0.312 and MUSIQ 40.85, beating SPIn-NeRF and Sparf on these metrics, though its KID (0.065) is worse than SPIn-NeRF's 0.042.
- Ablation on LLFF shows LPIPS improving from 0.323 (BaseNeRF) to 0.269 with fine-tuned inpainting and enhancement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ExtraNeRF instead of the strongest listed baseline | LLFF view extrapolation · LLFF view extrapolation | PSNR: 20.56 -> 20.76 | Helped |
| Use ExtraNeRF instead of the strongest listed baseline | LLFF view extrapolation · LLFF view extrapolation | SSIM: 0.683 -> 0.688 | Helped |
| Use ExtraNeRF instead of the strongest listed baseline | LLFF view extrapolation · LLFF view extrapolation | LPIPS: 0.284 -> 0.269 | Helped |
| Use ExtraNeRF instead of the strongest listed baseline | LLFF view extrapolation · LLFF view extrapolation | KID: 0.0156 -> 0.0154 | Helped |
| Use ExtraNeRF instead of the strongest listed baseline | LLFF view extrapolation · LLFF view extrapolation | MUSIQ: 51.84 -> 54.13 | Helped |
| Use ExtraNeRF instead of the strongest listed baseline | LLFF few-shot view synthesis · LLFF few-shot view synthesis | PSNR: 20.20 -> 21.17 | Helped |
| Use ExtraNeRF instead of the strongest listed baseline | LLFF few-shot view synthesis · LLFF few-shot view synthesis | SSIM: 0.630 -> 0.719 | Helped |
| Use ExtraNeRF instead of the strongest listed baseline | LLFF few-shot view synthesis · LLFF few-shot view synthesis | LPIPS: 0.308 -> 0.264 | Helped |
| Use ExtraNeRF instead of the strongest listed baseline | Tanks & Temples · Tanks & Temples | PSNR: 19.21 -> 19.20 | Hurt |
| Use ExtraNeRF instead of the strongest listed baseline | Tanks & Temples · Tanks & Temples | SSIM: 0.683 -> 0.718 | Helped |
| Use ExtraNeRF instead of the strongest listed baseline | Tanks & Temples · Tanks & Temples | LPIPS: 0.348 -> 0.312 | Helped |
| Use ExtraNeRF instead of the strongest listed baseline | Tanks & Temples · Tanks & Temples | KID: 0.042 -> 0.065 | Hurt |
Surprising findings
- Pretrained inpainting without per-scene fine-tuning yields blurry, color-drifted NeRF renderings, so scene-specific fine-tuning is essential.
- On Tanks & Temples the method has a worse KID score than SPIn-NeRF, which the authors attribute to the small dataset size.
Limitations
- Relies on per-scene fine-tuning and iterative optimization, adding hours of compute per scene.
- Depth maps require a 6-hour Sparf run as the bottleneck step.
- Evaluated mainly on LLFF and a small Tanks & Temples subset, with KID not uniformly best.
Experimental setup
Datasets: LLFF Tanks & Temples Places2
Benchmarks: LLFF Tanks & Temples
Metrics: PSNR SSIM LPIPS KID MUSIQ
Baselines: Sparf FreeNeRF DiffusioNeRF SinNeRF SPIn-NeRF SDS
Builds on: NeRF latent diffusion DreamBooth SPIn-NeRF Sparf
Closest papers by meaning
- Few-Shot Neural Radiance Fields under Unconstrained Illumination (AAAI 2024 · similarity 0.89)
- InsertNeRF: Instilling Generalizability into NeRF with HyperNet Modules (ICLR 2024 · similarity 0.88)
- SIGNeRF: Scene Integrated Generation for Neural Radiance Fields (CVPR 2024 · similarity 0.88)
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild (CVPR 2024 · similarity 0.88)
- Learning Neural Exposure Fields for View Synthesis (NeurIPS 2025 · similarity 0.88)
- In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.