Research map / 3D vision / NeRF and radiance fields
InNeRF360: Text-Guided 3D-Consistent Object Inpainting on 360-degree Neural Radiance Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Text-guided removal and inpainting of objects in 360-degree NeRFs using depth-warped SAM segmentation refinement plus 3D diffusion geometric priors and LPIPS-based appearance priors.
Problem
Removing text-specified objects from unbounded 360-degree NeRF scenes while producing view-consistent inpainting without floater artifacts; existing NeRF inpainting methods are limited to frontal-facing scenes.
Approach
Refines text-grounded SAM masks across views by warping rendered NeRF depth to generate point prompts, then retrains an inpainted NeRF with a 3D DDPM-based density score distillation geometric prior to remove floaters and a patch-based LPIPS appearance prior.
Key results
- On captured Cup and Starbucks scenes, the full method reaches LPIPS 0.5377 and FID 159.76 (Cup) versus 0.6149/201.70 for per-frame inpainting and 0.6421/252.34 for SPN-360.
- In a user study on view consistency, participants preferred InNeRF360 over per-frame inpainting in 71% to 92% of cases across five scenes.
- Qualitatively, depth-warping prompt refinement yields complete masks for transparent vases and partial slippers where Dino and SPN fail.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InNeRF360 instead of Per-Frame | 360-degree NeRF inpainting · Cup and Starbucks | LPIPS: 0.6149 -> 0.5377; 0.5981 -> 0.4523 | Helped |
| Use InNeRF360 instead of SPN-360 | 360-degree NeRF inpainting · Cup and Starbucks | FID: 252.34 -> 159.76; 215.28 -> 153.46 | Helped |
| Add Lin appearance prior (ablation) | Nerfacto · Cup and Starbucks | LPIPS: 0.7328 -> 0.7137; 0.6832 -> 0.6658 | Helped |
| Add Lgeom geometric prior (ablation) | Nerfacto · Cup and Starbucks | FID: 271.56 -> 189.57; 258.39 -> 166.45 | Helped |
| Add both Lin and Lgeom (ablation) | Nerfacto · Cup and Starbucks | LPIPS: 0.6197 -> 0.5377; 0.5795 -> 0.4523 | Helped |
| Add both Lin and Lgeom (ablation) | Nerfacto · Cup and Starbucks | FID: 189.57 -> 159.76; 166.45 -> 153.46 | Helped |
| Use InNeRF360 instead of per-frame inpainting | 360-degree NeRF inpainting · Garden, Room, Vasedeck, Bulldozer, Bear | user preference for temporal consistency: 89% vs 11%; 71% vs 29%; 81% vs 19%; 83% vs 17%; 92% vs 8% | Helped |
Limitations
- Performance depends on the 2D object detector; inaccurate or noisy initial boxes can break refinement.
- Inherits constraints of vision-language models for grounding text instructions.
- No ground truth segmentation masks for 360-degree scenes, so segmentation is evaluated only qualitatively.
Experimental setup
Datasets: MipNeRF-360 MipNeRF NeRFStudio scenes SPIn-NeRF IBRNet ShapeNet
Benchmarks: captured 360-degree scenes with and without object (Cup, Starbucks) user study on view consistency
Metrics: LPIPS FID user preference rate
Baselines: SPIn-NeRF segmentation (SPN) SPN-360 Dino video segmentation per-frame image inpainting Instruct-NeRF2NeRF
Builds on: Nerfacto SPIn-NeRF Segment Anything Model GLIP DSDS
Closest papers by meaning
- In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment (NeurIPS 2024 · similarity 0.88)
- Language-driven Object Fusion into Neural Radiance Fields with Pose-Conditioned Dataset Updates (CVPR 2024 · similarity 0.87)
- ExtraNeRF: Visibility-Aware View Extrapolation of Neural Radiance Fields with Diffusion Models (CVPR 2024 · similarity 0.86)
- SIGNeRF: Scene Integrated Generation for Neural Radiance Fields (CVPR 2024 · similarity 0.86)
- Global and Hierarchical Geometry Consistency Priors for Few-shot NeRFs in Indoor Scenes (CVPR 2024 · similarity 0.86)
- LAENeRF: Local Appearance Editing for Neural Radiance Fields (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.