Research map / 3D vision / NeRF and radiance fields
Language-driven Object Fusion into Neural Radiance Fields with Pose-Conditioned Dataset Updates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Inserts or removes objects in a background NeRF by fine-tuning Stable Diffusion inpainting on object/background images and updating the NeRF dataset with pose-ordered, diffusion-generated views.
Problem
Adding or removing objects in NeRF scenes in a view-consistent way, without requiring meshes, depth, masks, or semantics priors.
Approach
Fine-tune a text-to-image inpainting diffusion model on object and background images, then progressively update the NeRF's training set with diffusion-blended views ordered by camera pose proximity, using nearby NeRF renderings as color hints with low denoising strength.
Key results
- Ours achieves CLIPScore 0.2743 and CLIPDC 0.1642, the best among compared methods (inpainting 0.2683 / 0.1325; traditional 3D 0.2648 / 0.1169).
- Zero123 and Instruct-NeRF2NeRF score lowest on CLIPDC (0.0361 and 0.0263), indicating poor cross-view consistency of edits.
- Fine-tuning on object images is needed for the same object to appear across views, and pose-conditioned updates avoid inconsistent object poses seen with random view updates (qualitative ablation).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed pose-conditioned NeRF editing method instead of the strongest baseline | NeRF object insertion · 35 unique edits | CLIPScore: 0.2683 -> 0.2743 | Helped |
| Use the proposed pose-conditioned NeRF editing method instead of the strongest baseline | NeRF object insertion · 35 unique edits | CLIPDC: 0.1325 -> 0.1642 | Helped |
| Fine-tune the diffusion model on object images (ablation) | Object insertion pipeline · Multi-view object insertion | cross-view object consistency | Helped |
| Fine-tune the diffusion model on background images (ablation) | Object insertion pipeline · Multi-view object insertion | background preservation | Helped |
| Use pose-conditioned rather than random-pose dataset updates (ablation) | NeRF object insertion · Multi-view object insertion | cross-view pose consistency | Helped |
| Periodically replace old training views with new views (ablation) | NeRF object insertion · Multi-view object insertion | rendering consistency | Helped |
| Use pseudo ground-truth background images during removal (ablation) | NeRF object removal · Object removal | view consistency | Helped |
Surprising findings
- Instruct-NeRF2NeRF, a dataset-update method, fails at object insertion because it tends to preserve original geometry.
Limitations
- Views synthesized by a diffusion model may flicker, as in diffusion-based video editing.
- No theoretical foundation for pose-conditioned dataset updates is provided.
- Evaluation relies on CLIP-based metrics since no ground truth exists for scene editing.
Experimental setup
Datasets: self-captured multi-view object and background images Mip-NeRF-360 IBRNet NeRF synthetic DreamFusion synthetic data
Benchmarks: self-collected object insertion scenes (35 edits: 5 scenes x 7 objects) Mip-NeRF-360 and IBRNet (object removal)
Metrics: CLIPScore CLIP Directional Consistency (CLIPDC)
Baselines: Traditional 3D pipeline (COLMAP mesh + manual placement) Stable Diffusion inpainting Zero123 Instruct-NeRF2NeRF
Builds on: Stable Diffusion Instruct-NeRF2NeRF DreamBooth COLMAP Instant-NGP
Closest papers by meaning
- Removing Objects From Neural Radiance Fields (CVPR 2023 · similarity 0.92)
- D-NeRF: Neural Radiance Fields for Dynamic Scenes (CVPR 2021 · similarity 0.91)
- NeRF-Editing: Geometry Editing of Neural Radiance Fields (CVPR 2022 · similarity 0.91)
- SIGNeRF: Scene Integrated Generation for Neural Radiance Fields (CVPR 2024 · similarity 0.90)
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild (CVPR 2024 · similarity 0.89)
- InsertNeRF: Instilling Generalizability into NeRF with HyperNet Modules (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.