Research map / 3D vision / 3D generation
NeRFiller: Completing Scenes via Generative 3D Inpainting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Completes missing regions of 3D captures (NeRFs) by distilling multi-view-consistent 2D inpaints from an off-the-shelf Stable Diffusion inpainter, using a grid-prior joint sampling and iterative dataset updates.
Problem
Completing missing or occluded regions of 3D scenes and objects captured as NeRFs, where 2D inpainters produce multi-view inconsistent outputs and existing 3D inpainting work focuses on object removal with tight masks.
Approach
Discovers a "Grid Prior": tiling four views into a 2x2 grid for diffusion inpainting yields more 3D-consistent inpaints; generalizes it to many views via Joint Multi-View Inpainting (averaging noise predictions over random grid shuffles), and distills results into a NeRF with an annealed-noise Inpaint Iterative Dataset Update plus optional depth ranking loss.
Key results
- Joint Multi-View Inpainting reaches PSNR 15.89 / SSIM 0.82 / LPIPS 0.23 on NeRF synthetic consistency, the best among inpainting strategies (Grid Prior 14.43, SD Image Cond 14.15).
- On 10 completion scenes, Ours achieves PSNR 28.28 / SSIM 0.91 / LPIPS 0.06 versus 26.60 PSNR for Inpaint + DU and 27.39 for LaMask.
- Ours obtains 696 high-quality LoFTR correspondences, the highest among methods (Inpaint + DU 660, LaMask 643).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Joint Multi-View Inpainting instead of independent inpainting | Stable Diffusion inpainting · NeRF synthetic dataset (8 scenes) | PSNR: 14.15 -> 15.89 | Helped |
| Use Joint Multi-View Inpainting instead of independent inpainting | Stable Diffusion inpainting · NeRF synthetic dataset (8 scenes) | LPIPS: 0.28 -> 0.23 | Helped |
| Use NeRFiller instead of LaMask | 3D scene completion · 10 scene completion scenes | Corrs: 643 -> 696 | Helped |
| Use NeRFiller instead of LaMask | 3D scene completion · 10 scene completion scenes | PSNR: 27.39 -> 28.28 | Helped |
| Add relative depth ranking supervision (ablation) | NeRFiller · 10 scene completion scenes | Corrs: 682 -> 696 | Helped |
| Add relative depth ranking supervision (ablation) | NeRFiller · 10 scene completion scenes | PSNR: 28.41 -> 28.28 | No clear effect |
| Anneal the noise level from full noise to 0.4 rather than use a random noise schedule (ablation) | Inpaint DU · 3D scene completion | visual sharpness | Helped |
| Use depth ranking supervision (ablation) | NeRFiller on indoor scenes · Indoor scene completion | geometry quality | Helped |
Surprising findings
- LaMask scores high on NeRF consistency metrics mainly because it often copies the white background into the hole, producing a failed all-white NeRF.
Limitations
- Blurry or low-resolution detail far from training cameras due to downsampling into 2x2 grids.
- Struggles with casual captures with large masks; SD fails on those mask patterns without dilation.
- 2D-based approach, so 3D consistency is not guaranteed.
Experimental setup
Datasets: NeRF synthetic Objaverse Sketchfab SPIn-NeRF backpack scene
Benchmarks: NeRF synthetic (inpainting consistency) 10 custom 3D scene completion datasets
Metrics: PSNR SSIM LPIPS MUSIQ LoFTR correspondences (Corrs)
Baselines: Masked NeRF LaMa (LaMask) SD Text Cond SD Image Cond Extended Attention Inpaint + DU (IN2N-style)
Builds on: Stable Diffusion inpainting Instruct-NeRF2NeRF dataset update MultiDiffusion SPIn-NeRF Extended attention (Tune-A-Video)
Closest papers by meaning
- MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior (CVPR 2024 · similarity 0.89)
- Instant3dit: Multiview Inpainting for Fast Editing of 3D Objects (CVPR 2025 · similarity 0.89)
- In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment (NeurIPS 2024 · similarity 0.89)
- InstaInpaint: Instant 3D-Scene Inpainting with Masked Large Reconstruction Model (NeurIPS 2025 · similarity 0.87)
- NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows (CVPR 2024 · similarity 0.87)
- MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.