Research map / 3D vision / NeRF and radiance fields
LAENeRF: Local Appearance Editing for Neural Radiance Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAENeRF, a palette-based NeRF-like module that learns a mapping from estimated ray terminations to colors to enable local recoloring and style transfer of selected regions in pre-trained NeRFs, distilled back into the NeRF.
Problem
Local appearance editing of pre-trained NeRFs is hard: existing methods cause background artefacts, lack interactivity, support only global changes, or need large memory and long compute, and none combine interactive recoloring with style transfer.
Approach
Learn a lightweight palette-based mapping (hash-grid encoded per-ray termination points, barycentric weights and view-dependent offsets over learnable base colors with additive layer mixing) supervised by content and optional VGG Gram style losses plus geometry-aware depth-guided TV and depth-discontinuity losses, then distill edits into the NeRF using alpha-blended target colors, with a growing-q
Key results
- Reduces background MSE on LLFF recoloring by 59% relative to PaletteNeRF with semantic guidance (average 0.0007 vs 0.0017 for PaletteNeRF semantic in Table 1 scenes).
- On NeRF-Synthetic and LLFF stylization, background MSE averages 0.0048 vs 0.0270 for Ref-NPR.
- Recoloring of a selected object takes 3 min vs 13.5 min for PaletteNeRF on the LLFF Flower scene; previews appear after about 20 s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LAENeRF instead of PaletteNeRF for local recoloring | NeRF appearance editing · LLFF background | MSE: 0.0017 -> 0.0007 | Helped |
| Use LAENeRF instead of ICE-NeRF for local recoloring | NeRF appearance editing · LLFF background | MSE: 0.0075 -> 0.0007 | Helped |
| Use LAENeRF instead of PaletteNeRF for recoloring | NeRF appearance editing · mip-NeRF 360 indoor background | MSE: 0.0033 -> 0.0016 | Helped |
| Use LAENeRF instead of PaletteNeRF trained without recoloring | NeRF appearance editing · mip-NeRF 360 indoor background | MSE: 0.0018 -> 0.0016 | Helped |
| Use LAENeRF instead of Ref-NPR for stylization | NeRF appearance editing · NeRF-Synthetic background | MSE: 0.0466 -> 0.0071 | Helped |
| Use LAENeRF instead of Ref-NPR for stylization | NeRF appearance editing · LLFF background | MSE: 0.0073 -> 0.0025 | Helped |
| Use LAENeRF instead of PaletteNeRF | LLFF Flower recoloring · LLFF Flower | time: 13.5 min -> 3 min | Helped |
| Use LAENeRF instead of Ref-NPR | scene stylization · scene stylization | time: 2.5 min -> 2 min | Helped |
Surprising findings
- On mip-NeRF 360 indoor scenes, LAENeRF after recoloring has lower background MSE than non-recolored PaletteNeRF.
Limitations
- Stylized regions can only be modified by adjusting palette bases after training.
- Non-Lambertian effects modeled behind surfaces can reduce edit quality on real-world scenes.
- Separation of optimization and distillation adds time for dataset generation and NeRF fine-tuning.
Experimental setup
Datasets: NeRF-Synthetic LLFF mip-NeRF 360
Benchmarks: LLFF mip-NeRF 360 NeRF-Synthetic
Metrics: MSE in background user preference view-consistency rating runtime
Baselines: PaletteNeRF ICE-NeRF Ref-NPR
Builds on: NeRF Instant-NGP NeRFShop PaletteNeRF Ref-NPR
Closest papers by meaning
- PaletteNeRF: Palette-Based Appearance Editing of Neural Radiance Fields (CVPR 2023 · similarity 0.93)
- IReNe: Instant Recoloring of Neural Radiance Fields (CVPR 2024 · similarity 0.89)
- InsertNeRF: Instilling Generalizability into NeRF with HyperNet Modules (ICLR 2024 · similarity 0.89)
- Rad-NeRF: Ray-decoupled Training of Neural Radiance Field (NeurIPS 2024 · similarity 0.88)
- Few-Shot Neural Radiance Fields under Unconstrained Illumination (AAAI 2024 · similarity 0.88)
- SpectralNeRF: Physically Based Spectral Rendering with Neural Radiance Field (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.