Research map / 3D vision / NeRF and radiance fields
SIGNeRF: Scene Integrated Generation for Neural Radiance Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SIGNeRF, a modular pipeline that edits or inserts objects into existing NeRF scenes by generating a depth-conditioned ControlNet reference sheet (grid of views) and then updating the NeRF image dataset in one pass before fine-tuning.
Problem
Editing existing photorealistic NeRF scenes with generative 2D diffusion models is hard because per-view generations are inconsistent across views, and prior methods (SDS-based, iterative dataset updates) are slow and lack fine-grained spatial control.
Approach
Arrange multiple views (RGB, depth, mask) into a single image grid so depth-conditioned ControlNet inpainting produces multi-view consistent edits in one generation; a reference sheet with one empty slot is then used to propagate the edit to every training image, with spatial control via bounding-box or proxy mesh selection.
Key results
- Achieves CLIP text-to-image directional similarity of 0.2125 versus 0.1600 for Instruct-NeRF2NeRF and 0.1849 for DreamEditor (averaged over 10 scenes).
- Background preservation PSNR 32.22 and SSIM 0.83, compared with 30.09 and 0.64 for Instruct-NeRF2NeRF.
- Takes about half the time of Instruct-NeRF2NeRF and DreamEditor (which need over an hour) on a single GPU with one dataset iteration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SIGNeRF instead of Instruct-NeRF2NeRF | NeRF scene editing · 10 scenes | CLIP text-to-image directional similarity (Image): 0.1603 -> 0.23 | Helped |
| Use SIGNeRF instead of Instruct-NeRF2NeRF | NeRF scene editing · 10 scenes | CLIP text-to-image directional similarity (NeRF): 0.1600 -> 0.2125 | Helped |
| Use SIGNeRF instead of DreamEditor | NeRF scene editing · 10 scenes | CLIP text-to-image directional similarity (NeRF): 0.1849 -> 0.2125 | Helped |
| Use SIGNeRF instead of Instruct-NeRF2NeRF | NeRF scene editing · 10 scenes | background preservation PSNR: 30.09 -> 32.22 | Helped |
| Use SIGNeRF instead of Instruct-NeRF2NeRF | NeRF scene editing · 10 scenes | background preservation SSIM: 0.64 -> 0.83 | Helped |
Surprising findings
- Generating a grid of views in one ControlNet pass yields much better multi-view consistency than generating each view individually with the same seed and parameters.
Limitations
- Images must be downscaled to fit the reference sheet, losing quality in the generated edit.
- Off-center or distant objects reduce latent resolution and edit quality; off-center objects make consistent sheets hard.
- Unsuitable for extended scene modifications such as background editing, which produces strong inconsistencies.
Experimental setup
Datasets: Instruct-NeRF2NeRF scenes DTU BlendedMVS custom smartphone/drone captures (PolyCam, COLMAP)
Benchmarks: Instruct-NeRF2NeRF scenes DTU BlendedMVS custom scenes
Metrics: CLIP text-to-image directional similarity PSNR SSIM
Baselines: Instruct-NeRF2NeRF DreamEditor
Builds on: ControlNet Nerfstudio Instruct-NeRF2NeRF SDXL Nerfacto
Closest papers by meaning
- Language-driven Object Fusion into Neural Radiance Fields with Pose-Conditioned Dataset Updates (CVPR 2024 · similarity 0.90)
- NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion (ICML 2023 · similarity 0.90)
- ExtraNeRF: Visibility-Aware View Extrapolation of Neural Radiance Fields with Diffusion Models (CVPR 2024 · similarity 0.88)
- Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation (CVPR 2024 · similarity 0.88)
- Enhancing Neural Radiance Fields with Adaptive Multi-Exposure Fusion: A Bilevel Optimization Approach for Novel View Synthesis (AAAI 2024 · similarity 0.88)
- InsertNeRF: Instilling Generalizability into NeRF with HyperNet Modules (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.