Research map / 3D vision / NeRF and radiance fields
FFaceNeRF: Few-shot Face Editing in Neural Radiance Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FFaceNeRF, a few-shot, mask-based 3D face editing method that adds a geometry adapter with tri-plane feature injection, latent mixing augmentation, and an overlap (DICE) optimization on top of NeRFFaceEditing/EG3D, needing only about 10 training masks.
Problem
Mask-based 3D-aware face editing with NeRFs is limited to the fixed segmentation layouts of pretrained segmentation networks, and training new layouts needs large datasets.
Approach
A lightweight geometry adapter, fed with injected tri-plane features and view direction, is trained on ~10 user-defined masks while the rest of the pretrained EG3D/NeRFFaceEditing is frozen; latent mixing of late style layers augments tri-planes without changing semantics, and a DICE-based overlap loss improves small-region editing.
Key results
- Mask generation mIoU 85.33% vs 81.37% for NeRFFaceEditing on 22 test masks.
- Perceptual A/B study with 21 participants: ours preferred over NeRFFaceEditing at 72.29% faithfulness, 67.83% retention, 68.68% quality; over IDE-3D at 79.65%, 80.17%, 81.22%.
- Ablation with 10 training samples: full model mIoU 0.850 vs 0.835 without feature injection and 0.845 without LMTA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use geometry-adapter feature injection (ablation) | FFaceNeRF · Base layout, 22 test masks; 5 training samples | mIoU: 0.806 vs 0.832 | Helped |
| Use LMTA augmentation (ablation) | FFaceNeRF · Base layout, 22 test masks; 1 training sample | mIoU: 0.695 vs 0.711 | Helped |
| Use LMTA augmentation (ablation) | FFaceNeRF · Base layout, 22 test masks; 5 training samples | mIoU: 0.829 vs 0.832 | Helped |
| Mix all latent layers instead of selected layers (ablation) | FFaceNeRF · Base layout, 22 test masks; 5 training samples | mIoU: 0.654 vs 0.832 | Hurt |
| Use overlap-based rather than percentage-based optimization | FFaceNeRF · Eye editing | Target-mask fidelity | Helped |
| Use FFaceNeRF rather than NeRFFaceEditing | 3D-aware face mask generation · Base layout, 22 test masks | mIoU (%): 85.33 vs 81.37 | Helped |
| Use FFaceNeRF rather than IDE-3D | Face editing · Perceptual study | Quality preference: 81.22 | Helped |
Surprising findings
- Mixing all latent layers severely degrades mask accuracy (mIoU 0.603 with one sample, 0.780 with 30), so only late layers are safe for augmentation.
- Mixing layers ranked by highest L1 change gave consistently low mIoU (<0.8), making them unsuitable for local editing.
Limitations
- Inference is iterative optimization taking around 31 seconds per edit, not real-time.
- One-shot setting is limited because the geometry adapter cannot generalize from a single sample.
Experimental setup
Datasets: CelebA-HQ FFHQ-style randomly generated images with annotated masks (Base, Eyes, Nose layouts)
Benchmarks: 22 annotated Base-layout test masks CelebA-HQ qualitative editing
Metrics: mIoU L1 distance A/B preference percentage
Baselines: NeRFFaceEditing IDE-3D DatasetGAN percentage-based optimization
Builds on: EG3D NeRFFaceEditing pivotal tuning inversion DatasetGAN
Closest papers by meaning
- StrokeFaceNeRF: Stroke-based Facial Appearance Editing in Neural Radiance Field (CVPR 2024 · similarity 0.90)
- InstructPix2NeRF: Instructed 3D Portrait Editing from a Single Image (ICLR 2024 · similarity 0.87)
- LAENeRF: Local Appearance Editing for Neural Radiance Fields (CVPR 2024 · similarity 0.86)
- SIGNeRF: Scene Integrated Generation for Neural Radiance Fields (CVPR 2024 · similarity 0.85)
- 3D-Aware Face Editing via Warping-Guided Latent Direction Learning (CVPR 2024 · similarity 0.85)
- InsertNeRF: Instilling Generalizability into NeRF with HyperNet Modules (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.