Research map / 3D vision / 3D generation
MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MVIP-NeRF, which inpaints NeRF scenes by distilling a Stable Diffusion inpainting prior via RGB SDS, normal-map SDS, and a multi-view SDS score, without explicit 2D RGB or depth inpainting supervision.
Problem
NeRF inpainting relies on explicit RGB and depth outputs from 2D inpainters, which produce view-inconsistent appearance and poorly aligned or low-quality geometry in masked regions, especially under large view variations.
Approach
Replace explicit 2D RGB and depth inpainting supervision with iterative score distillation from a latent diffusion inpainting model applied to rendered RGB and normal maps, plus a multi-view SDS term that shares the noise and timestep across several nearby views to enforce cross-view consistency.
Key results
- On Real-S, MVIP-NeRF reaches LPIPS 0.507 versus 0.574 for SPIn-NeRF+LaMa and 0.665 for Remove-NeRF+LaMa, with PSNR comparable (17.667).
- On Real-L, MVIP-NeRF attains depth L2 0.021 versus 0.038 (Remove-NeRF) and 0.090 (SPIn-NeRF), while LPIPS is 0.181 vs 0.187 for Remove-NeRF.
- On Real-L, Remove-NeRF gets better LPIPS (0.187) than SPIn-NeRF (0.215) and MVIP-NeRF's FID (100.452) is slightly worse than Remove-NeRF's 88.245.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MVIP-NeRF instead of the strongest baseline | NeRF inpainting · Real-S | LPIPS: 0.574 -> 0.507 | Helped |
| Use MVIP-NeRF instead of the strongest baseline | NeRF inpainting · Real-L | Depth L2: 0.038 -> 0.021 | Helped |
| Add appearance diffusion to Masked NeRF (ablation) | MVIP-NeRF · Real-S | LPIPS: 0.761 -> 0.587 | Helped |
| Add appearance diffusion to Masked NeRF (ablation) | MVIP-NeRF · Real-L | LPIPS: 0.222 -> 0.191 | Helped |
| Use explicit inpainted depth instead of geometry diffusion (ablation) | MVIP-NeRF · Real-L | LPIPS: 0.195 -> 0.254 | Hurt |
| Replace explicit inpainted depth with geometry diffusion (ablation) | MVIP-NeRF · Real-L | Depth L2: 0.044 -> 0.031 | Helped |
| Add multi-view score distillation (ablation) | MVIP-NeRF · Real-S | LPIPS: 0.556 -> 0.507 | Helped |
| Add multi-view score distillation (ablation) | MVIP-NeRF · Real-L | Depth L2: 0.031 -> 0.021 | Helped |
Surprising findings
- Remove-NeRF beats SPIn-NeRF on LPIPS in the Real-L setting, which the authors attribute to its view selection avoiding incorrect 2D inpaintings.
- Replacing the diffusion prior with a CLIP loss yields weak recovery of appearance and geometry.
Limitations
- Iterative diffusion-prior optimization is slow.
- Requires tuning of classifier-free guidance hyperparameters.
- Cannot remove shadows.
Experimental setup
Datasets: Real-S (10 scenes from SPIn-NeRF) Real-L (16 scenes from Remove-NeRF)
Benchmarks: Real-S Real-L
Metrics: PSNR LPIPS FID depth L2
Baselines: SPIn-NeRF Remove-NeRF LaMa
Builds on: SDS (DreamFusion) SPIn-NeRF Stable Diffusion inpainting NeRF
Closest papers by meaning
- NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows (CVPR 2024 · similarity 0.90)
- MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing (NeurIPS 2024 · similarity 0.89)
- NeRFiller: Completing Scenes via Generative 3D Inpainting (CVPR 2024 · similarity 0.89)
- In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment (NeurIPS 2024 · similarity 0.89)
- NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs (CVPR 2024 · similarity 0.87)
- Towards Enhanced Image Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistency (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.