Research map / Image generation / Editing and style transfer
ED-NeRF: Efficient Text-Guided Editing of 3D Scene With Latent Space NeRF
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ED-NeRF, which trains a TensoRF NeRF in Stable Diffusion's latent space with a refinement layer and edits it via masked Delta Denoising Score distillation plus masked reconstruction regularization.
Problem
Text-guided editing of real-world 3D NeRF scenes is slow and poorly edit-friendly; existing CLIP-based and SDS-based methods require full-resolution rendering or distort untargeted regions.
Approach
Embed real-scene multi-view VAE latents into a latent-space NeRF with a learned refinement layer (ResNet and self-attention) to recover fidelity, then extend Delta Denoising Score to 3D with a text-derived binary mask and masked reconstruction loss so edits stay local.
Key results
- CLIP directional score of 0.2265 for ED-NeRF vs 0.2053 for Instruct N2N and 0.1947 for NeRF-ART.
- Fine-tuning takes 14 min on RTX 3090 with 8GB GPU memory, versus 90 min and 15GB for Instruct N2N.
- User study preservation score 4.09 vs 3.08 for Instruct N2N; text score 3.88 vs 3.29.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ED-NeRF instead of the strongest listed baseline, Instruct N2N | 3D scene editing · 3D scene editing | CLIP Direction Score: 0.2053 -> 0.2265 | Helped |
| Use ED-NeRF instead of the strongest listed baseline, Instruct N2N | 3D scene editing · 3D scene editing | Text score: 3.29 -> 3.88 | Helped |
| Use ED-NeRF instead of the strongest listed baseline, Instruct N2N | 3D scene editing · 3D scene editing | Preservation: 3.08 -> 4.09 | Helped |
| Use ED-NeRF instead of the best baseline, NeRF-Art | 3D scene editing · 3D scene editing | view consistency: 3.79 -> 3.64 | Hurt |
| Use ED-NeRF instead of Instruct N2N | 3D scene editing · 3D scene editing | Fine-tuning time: 90min -> 14min | Helped |
| Use ED-NeRF instead of the lowest-memory baseline, Instruct N2N | 3D scene editing · 3D scene editing | GPU Memory: 15GB -> 8GB | Helped |
| Add masked reconstruction regularization to DDS editing (ablation) | ED-NeRF · 3D scene editing | untargeted-region preservation | Helped |
| Add mask guidance to DDS (ablation) | ED-NeRF · 3D scene editing | untargeted-region preservation | Helped |
| Add the refinement layer (ablation) | ED-NeRF · 3D scene editing | reconstruction quality | Helped |
| Add masked reconstruction regularization to DDS editing (ablation) | ED-NeRF · 3D scene editing | target-text adherence | Helped |
| Replace the proposed refinement layer with MLP or ResNet (ablation) | ED-NeRF · Novel view synthesis | high-frequency detail quality | Hurt |
| Use different text-prompt segmentation models to generate masks (ablation) | ED-NeRF · 3D scene editing | editing consistency | No clear effect |
Surprising findings
- Naively matching NeRF to the VAE latent map degrades view synthesis, requiring the refinement layer.
- Latent-space rendering at 64x64 gave 2.4x faster RGB inference than TensoRF at 512 resolution.
Limitations
- Quality degrades when target text is very far from the source object (e.g., statue to black hole under a statue).
- Inherits limitations of the 2D diffusion model, causing failures in 3D editing cases.
- Evaluation is small: a user study with 20 subjects on 5 scenes and a CLIP directional score.
Experimental setup
Datasets: LLFF IBRNet InstructNeRF2NeRF face dataset
Benchmarks: LLFF IBRNet
Metrics: CLIP directional score user study score (text score, preservation, view consistency fine-tuning time GPU memory
Baselines: CLIP-NeRF NeRF-ART Instruct-NeRF2NeRF Masked SDS
Builds on: NeRF TensoRF Latent-NeRF Delta Denoising Score DreamFusion
Closest papers by meaning
- GenN2N: Generative NeRF2NeRF Translation (CVPR 2024 · similarity 0.89)
- Perturb-and-Revise: Flexible 3D Editing with Generative Trajectories (CVPR 2025 · similarity 0.89)
- DreamCatalyst: Fast and High-Quality 3D Editing via Controlling Editability and Identity Preservation (ICLR 2025 · similarity 0.88)
- Customize your NeRF: Adaptive Source Driven 3D Scene Editing via Local-Global Iterative Training (CVPR 2024 · similarity 0.88)
- NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows (CVPR 2024 · similarity 0.87)
- HIFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.