Research map / 3D vision / Face and shape
Learning Dense Correspondence for NeRF-Based Face Reenactment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PlaneDict, a learnable orthogonal plane-basis dictionary that maps motion codes to tri-plane deformations, enabling one-shot multi-view NeRF face reenactment without a 3DMM prior.
Problem
One-shot face reenactment requires dense correspondence between different NeRF-based face representations, which implicit representations lack, and existing approaches rely on 3DMM priors that limit non-facial regions and identity fidelity.
Approach
Decompose EG3D-style tri-planes into canonical tri-planes, identity deformations (StyleGAN generator), and motion deformations produced by a PlaneDict module that linearly combines learnable channel-wise orthogonal plane bases weighted by motion-generated factors, trained self-supervised without 3DMM.
Key results
- Self-reenactment: SSIM 0.870, PSNR 22.1, LPIPS 0.079, best among compared methods including HiDeNeRF (0.862/21.9/0.084).
- Cross-identity on VoxCeleb1: CSIM 0.911 vs HiDeNeRF 0.876; FID 49 vs 57; ET 5.18 vs 7.02.
- Multi-view synthesis vs HiDeNeRF: CSIM 0.840 vs 0.829, AVD 0.008 vs 0.014; ablation shows 20 plane bases chosen as performance plateaus.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PlaneDict instead of removing it (ablation) | Ours · Ablation study | CSIM: 0.763 -> 0.911 | Helped |
| use PlaneDict instead of removing it (ablation) | Ours · Ablation study | ET: 7.58 -> 5.18 | Helped |
| increase PlaneDict bases from 5 to 10 (ablation) | Ours · Ablation study | CSIM: 0.679 -> 0.802 | Helped |
| increase PlaneDict bases from 10 to 15 (ablation) | Ours · Ablation study | CSIM: 0.802 -> 0.899 | Helped |
| increase PlaneDict bases from 15 to 20 (ablation) | Ours · Ablation study | CSIM: 0.899 -> 0.911 | Helped |
| increase PlaneDict bases from 5 to 10 (ablation) | Ours · Ablation study | ET: 9.92 -> 7.36 | Helped |
| increase PlaneDict bases from 10 to 15 (ablation) | Ours · Ablation study | ET: 7.36 -> 6.69 | Helped |
| increase PlaneDict bases from 15 to 20 (ablation) | Ours · Ablation study | ET: 6.69 -> 5.18 | Helped |
| use Ours rather than HiDeNeRF | face reenactment · VoxCeleb1 cross-identity | CSIM: 0.876 -> 0.911 | Helped |
| use Ours rather than HiDeNeRF | face reenactment · VoxCeleb1 cross-identity | ET: 7.02 -> 5.18 | Helped |
| use Ours rather than HiDeNeRF | face reenactment · VoxCeleb2 cross-identity | CSIM: 0.787 -> 0.790 | Helped |
| use Ours rather than HiDeNeRF | face reenactment · TalkingHead-1KH cross-identity | FID: 52 -> 49 | Helped |
Surprising findings
- Using PlaneDict with fewer bases (5 or 10) performs worse than without PlaneDict on several metrics, indicating a basis-count threshold effect.
Limitations
- Cannot handle extreme poses and expressions due to dataset biases.
- Limited by hardware to a maximum of 23 plane bases in ablation.
Experimental setup
Datasets: VoxCeleb1 VoxCeleb2 TalkingHead-1KH FFHQ
Benchmarks: VoxCeleb1 VoxCeleb2 TalkingHead-1KH
Metrics: SSIM PSNR LPIPS FID CSIM AUCON PRMSE AVD ET
Baselines: FOMM DaGAN ROME FNeVR HiDeNeRF
Builds on: EG3D StyleGAN HiDeNeRF NeRF
Closest papers by meaning
- A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (NeurIPS 2021 · similarity 0.87)
- Joint Optimization of Neural Radiance Fields and Continuous Camera Motion from a Monocular Video (CVPR 2025 · similarity 0.87)
- NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs (CVPR 2024 · similarity 0.86)
- Mask-Based Modeling for Neural Radiance Fields (ICLR 2024 · similarity 0.86)
- CF-NeRF: Camera Parameter Free Neural Radiance Fields with Incremental Learning (AAAI 2024 · similarity 0.86)
- VOODOO 3D: Volumetric Portrait Disentanglement For One-Shot 3D Head Reenactment (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.