Research map / 3D vision / 3D generation
In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes In-N-Out, which inpaints one sampled view with Stable Diffusion and propagates it to other views via NeRF-based initial-latent alignment and cross-view attention, then optimizes NeRF with a patch-based LPIPS+adversarial loss for 3D object removal.
Problem
3D object removal from NeRFs is hard because multi-view supervision is missing for occluded regions, and 2D diffusion inpainters produce view-inconsistent outputs when applied to multi-view data.
Approach
Align diffusion inpainting across views explicitly by volume-rendering the initial noise latents through a pretrained NeRF (explicit latent alignment) and implicitly by swapping self-attention keys/values with those of a prior view during denoising (implicit latent alignment), with no diffusion fine-tuning.
Key results
- Achieves LPIPS 0.49, FID 130.92, MUSIQ 50.97, versus InFusion's 0.62 / 153.77 / 39.29 and SPIn-NeRF's 0.54 / 185.63 / 38.69.
- User study preference: 50.68% overall for the proposed method vs. 25.00% for InFusion and 21.82% for SPIn-NeRF.
- Ablation: removing the patch loss raises FID from 130.92 to 293.32; removing ELA or ILA raises FID to 133.09 and 141.78 respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add explicit latent alignment (ELA) (ablation) | In-N-Out · SPIn-NeRF | LPIPS: 0.52 -> 0.49 | Helped |
| add implicit latent alignment (ILA) (ablation) | In-N-Out · SPIn-NeRF | FID: 141.78 -> 130.92 | Helped |
| add patch-based hybrid loss (ablation) | In-N-Out · SPIn-NeRF | FID: 293.32 -> 130.92 | Helped |
| add LPIPS perceptual loss to patch loss (ablation) | In-N-Out · SPIn-NeRF | FID: 223.31 -> 130.92 | Helped |
| add adversarial loss to patch loss (ablation) | In-N-Out · SPIn-NeRF | MUSIQ: 49.88 -> 50.97 | Helped |
Surprising findings
- Sensitivity analysis shows the cross-view attention weight λa has little effect, which the authors attribute to explicit initial-latent alignment dominating the structure.
Limitations
- Struggles with full 3D consistency on high-frequency details due to 2D diffusion priors on multi-view data.
- Tailored to forward-facing scenes and not applicable to 360-degree views.
- Requires predefined object masks.
Experimental setup
Datasets: SPIn-NeRF dataset
Benchmarks: SPIn-NeRF dataset self-collected forward-facing scenes
Metrics: LPIPS FID MUSIQ user study preference
Baselines: SPIn-NeRF NeRFiller InFusion
Builds on: NeRF Stable Diffusion inpainting SPIn-NeRF NeRFiller DDIM nerfacto
Closest papers by meaning
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.89)
- MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior (CVPR 2024 · similarity 0.89)
- Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.89)
- NeRFiller: Completing Scenes via Generative 3D Inpainting (CVPR 2024 · similarity 0.89)
- MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes (CVPR 2025 · similarity 0.88)
- InNeRF360: Text-Guided 3D-Consistent Object Inpainting on 360-degree Neural Radiance Fields (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.