Research map / 3D vision / 3D generation
Customize your NeRF: Adaptive Source Driven 3D Scene Editing via Local-Global Iterative Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CustomNeRF, a unified framework for adaptive source-driven 3D scene editing from text or reference images, using Local-Global Iterative Editing and class-guided regularization on Instant-NGP with SDS-based diffusion guidance.
Problem
Editing real-world NeRF scenes so that only foreground regions change, conforming to a text prompt or a single reference image, while keeping backgrounds and maintaining multi-view consistency (Janus problem).
Approach
A foreground-aware NeRF with an extra editing-probability field lets training alternate between local SDS on foreground-only renders (with class-word prompts to reduce Janus) and global SDS on full images, plus a background color-preservation MSE loss; Custom Diffusion embeds reference subjects as a special token.
Key results
- Text-driven editing: CLIPdir 22.66 for CustomNeRF vs 16.34 for RePaint-NeRF and 11.48 for Instruct-NeRF2NeRF; 85.2% user vote.
- Image-driven editing: CLIPdir 20.07 and DINO similarity 47.44, vs 16.46 CLIPdir for RePaint-NeRF* and 15.47 for Ours+Splicing Loss; 75.6% user vote.
- Training takes 10 minutes to 1 hour on a single NVIDIA RTX 3090 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CustomNeRF instead of Ours+Splicing Loss | Image-driven editing · Image-driven editing | CLIP Text-Image Directional Similarity: 15.47 -> 20.07 | Helped |
| Use CustomNeRF instead of RePaint-NeRF* | Image-driven editing · Image-driven editing | DINO Similarity: 38.74 -> 47.44 | Helped |
| Use CustomNeRF instead of Ours+Splicing Loss | Image-driven editing · Image-driven editing | Vote percentage: 10.6% -> 75.6% | Helped |
| Use CustomNeRF instead of RePaint-NeRF | Text-driven editing · Text-driven editing | CLIP Text-Image Directional Similarity: 16.34 -> 22.66 | Helped |
| Use CustomNeRF instead of Instruct-NeRF2NeRF | Text-driven editing · Text-driven editing | Vote percentage: 10.6% -> 85.2% | Helped |
| Alternate local foreground editing with global full-image editing instead of using local editing only (ablation) | CustomNeRF LGIE · Text-driven editing | Background preservation and foreground-background harmony | Helped |
| Alternate local foreground editing with global full-image editing instead of using global editing only (ablation) | CustomNeRF LGIE · Text-driven editing | Background preservation and foreground-background harmony | Helped |
| Add class-guided regularization in the local editing stage by removing V* from the local prompt (ablation) | CustomNeRF image-driven editing · Image-driven editing | Multi-view consistency | Helped |
| Remove class-guided regularization from both local and global stages (ablation) | CustomNeRF image-driven editing · Image-driven editing | Multi-view consistency | Hurt |
| Apply class-guided regularization in the global stage by removing V* from the global prompt (ablation) | CustomNeRF image-driven editing · Image-driven editing | Multi-view consistency | Hurt |
Surprising findings
- Applying class priors in the global stage leads to an obvious Janus problem, suggesting class priors work better in simpler backgrounds.
- Ours+Splicing Loss achieves a relatively high DINO similarity largely because it optimizes DINO-ViT features, despite being qualitatively inferior.
Limitations
- Custom Diffusion cannot perfectly replicate reference images, causing detail inconsistencies in image-driven edits.
- Inherits oversaturation issues from DreamFusion.
- Limited to text and image editing prompts; no audio or sketch inputs.
Experimental setup
Datasets: BlendedMVS LLFF IBRNet Bear Statue
Benchmarks: BlendedMVS LLFF IBRNet Bear Statue
Metrics: CLIP text-image directional similarity DINO similarity user vote percentage
Baselines: Instruct-NeRF2NeRF RePaint-NeRF Ours+Splice Loss RePaint-NeRF*
Builds on: Instant-NGP Custom Diffusion DreamFusion Instruct-NeRF2NeRF RePaint-NeRF
Closest papers by meaning
- CTRL-D: Controllable Dynamic 3D Scene Editing with Personalized 2D Diffusion (CVPR 2025 · similarity 0.88)
- ED-NeRF: Efficient Text-Guided Editing of 3D Scene With Latent Space NeRF (ICLR 2024 · similarity 0.88)
- Improving Editability in Image Generation with Layer-wise Memory (CVPR 2025 · similarity 0.87)
- ProEdit: Simple Progression is All You Need for High-Quality 3D Scene Editing (NeurIPS 2024 · similarity 0.87)
- EditSplat: Multi-View Fusion and Attention-Guided Optimization for View-Consistent 3D Scene Editing with 3D Gaussian Splatting (CVPR 2025 · similarity 0.86)
- InstructPix2NeRF: Instructed 3D Portrait Editing from a Single Image (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.