Research map / Image generation / Editing and style transfer
AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AE-NeRF, a few-shot audio-driven talking head NeRF that uses audio-aware feature aggregation over reference images and a dual-NeRF split between audio-related and audio-independent face regions.
Problem
Few-shot audio-driven talking head synthesis with NeRFs suffers from weak facial priors (lack of multi-identity base model or audio-aware prior) and misalignment from modeling all face regions as audio-related.
Approach
An Audio Aware Aggregation module weights reference-image features by audio similarity to the target via cross attention, and an Audio-Aligned Face Generation strategy uses two NeRFs (audio-associated for lower face, audio-independent for the rest) with Regionwise Ray Sampling to cut training cost.
Key results
- On Testset A self-driven, AE-NeRF reaches PSNR 32.63, SSIM 0.949, LPIPS 0.078 and LMD 2.425, with SyncNet 6.904 versus DFRF's 5.219.
- With 15s training clips at 40k iterations, AE-NeRF attains SyncNet 6.217 versus DFRF 3.688 and AD-NeRF 3.528.
- Ablation: removing the Audio Aware Aggregation module drops SyncNet from 7.813 to 4.570 and raises LMD from 2.613 to 3.646.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AE-NeRF instead of DFRF | talking-head synthesis · Testset A | PSNR: 32.30 -> 32.63 | Helped |
| use AE-NeRF instead of DFRF | talking-head synthesis · Testset A | LMD: 3.023 -> 2.425 | Helped |
| use AE-NeRF instead of DFRF | talking-head synthesis · Testset B | Sync: 4.859 -> 6.217 | Helped |
| use AE-NeRF instead of DFRF | talking-head synthesis · Testset C | Sync: 5.321 -> 6.690 | Helped |
| add Audio Aware Aggregation rather than slot attention (ablation) | AE-NeRF · Ablation study | PSNR: 31.25 -> 32.64 | Helped |
| add Audio Aware Aggregation rather than slot attention (ablation) | AE-NeRF · Ablation study | Sync: 4.570 -> 7.813 | Helped |
| add Audio-Aligned Face Generation with separate audio-associated and audio-independent NeRFs (ablation) | AE-NeRF · Ablation study | LMD: 3.938 -> 2.613 | Helped |
| add the warper (ablation) | AE-NeRF · Ablation study | LMD: 2.779 -> 2.613 | Helped |
| add Regionwise Ray Sampling (ablation) | dual-NeRF AE-NeRF · Regionwise Ray Sampling study | Training Time / h: 50.6 -> 43.1 | Helped |
| vary overlap proportion ε from 0.2 to 0.4 (ablation) | AE-NeRF with Regionwise Ray Sampling · Regionwise Ray Sampling study | Sync: 7.268 -> 7.775 | Helped |
| vary overlap proportion ε from 0.4 to 0.6 (ablation) | AE-NeRF with Regionwise Ray Sampling · Regionwise Ray Sampling study | Sync: 7.775 -> 6.991 | Hurt |
| remove Regionwise Ray Sampling (ablation) | dual-NeRF AE-NeRF · Regionwise Ray Sampling study | Training Time / h: 43.1 -> 50.6 | Hurt |
Surprising findings
- AD-NeRF could not be made to generalize with a pre-trained base model, failing to produce plausible results.
- Wav2Lip achieves SyncNet scores above ground truth but with noticeably lower image quality.
Limitations
- Dual NeRF design slows training and inference relative to a single NeRF.
- Generalization to datasets with extreme poses and expressions is not addressed.
- Pose matrices are estimated with face2face, adding a preprocessing dependency.
Experimental setup
Datasets: AD-NeRF videos DFRF videos HDTF
Benchmarks: AD-NeRF videos (Testset A) DFRF videos HDTF (Testset B/C cross-driven)
Metrics: PSNR SSIM LPIPS LMD SyncNet confidence
Baselines: Wav2Lip ATVG MakeitTalk AD-NeRF DFRF
Builds on: NeRF DFRF AD-NeRF Wav2Vec
Closest papers by meaning
- SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis (CVPR 2024 · similarity 0.88)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.88)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.88)
- PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis (AAAI 2025 · similarity 0.88)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.87)
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.