Research map / Video / Video generation
Talking face generation
Generates audio-driven talking faces from single images with landmark priors and motion diffusion.
81 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Video generation, keyword cluster “audio · editing · talking”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 1 of 2 tracked benchmarks.
16 papers in 2023–24 and 50 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 7.0% to 4.5%.
Open questions the data points to
- HDTF (FID): best went from 9.057 (up to 2024) to 7.855 in 2025 — OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers (NeurIPS 2025 · 1 citation)
- Papers report Hallo on HDTF (FID) anywhere from 14.2 to 20.64: comparisons against it depend on whose number you take.
- Papers report Wav2Lip on HDTF (CPBD) anywhere from 0.299 to 0.79: comparisons against it depend on whose number you take.
No longer reported by papers here: MEAD (SSIM, last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Every Smile Is Unique: Landmark-Guided Diverse Smile Generation (CVPR 2018 · 60 citations)
- 2019 · Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss (CVPR 2019 · 440 citations)
- 2021 · One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (CVPR 2021 · 445 citations)
- 2022 · FaceFormer: Speech-Driven 3D Facial Animation With Transformers (CVPR 2022 · 252 citations)
- 2023 · Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert (CVPR 2023 · 109 citations)
- 2024 · VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · 40 citations)
- 2025 · Sonic: Shifting Focus to Global Audio Perception in Portrait Animation (CVPR 2025 · 12 citations)
- 2026 · PC-Talk: Precise Facial Animation Control for Audio-Driven Talking Face Generation (CVPR 2026 · 1 citation)
Earliest papers here
- Every Smile Is Unique: Landmark-Guided Diverse Smile Generation (CVPR 2018 · 60 citations)
- Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss (CVPR 2019 · 440 citations)
- Talking Face Generation by Adversarially Disentangled Audio-Visual Representation (AAAI 2019 · 428 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
HDTF · FID (lower is better)
9 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 9.057 | Say Anything with Any Style (AAAI 2024 · 11 citations) |
| 2025 | 7.855 | OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers (NeurIPS 2025 · 1 citation) |
MEAD · SSIM (higher is better)
4 papers, 2021–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 0.71 | Audio-Driven Emotional Video Portraits (CVPR 2021 · 206 citations) |
| 2023 | 0.837 | StyleTalk: One-Shot Talking Head Generation with Controllable Speaking Styles (AAAI 2023 · 75 citations) |
| 2024 | 0.88 | FG-EmoTalk: Talking Head Video Generation with Fine-Grained Controllable Facial Expressions (AAAI 2024 · 9 citations) |
What has been tried
55 papers here report 622 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 3%, hurt 23%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Train on 10% of the data instead of the full dataset | OneMin-32 · FVD_25 | 105.88 -> 147.401 | Hurt | VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · 40 citations) |
| replace TPS-based motion features with MRAA | PATS test set · FGD | 18.131 -> 288.378 | Hurt | Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model (CVPR 2024 · 15 citations) |
| Use N=8 rather than N=1 pseudo-word tokens | CelebV-HQ · Age | 0.23 -> 0.51 | Helped | FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio (CVPR 2024 · 15 citations) |
| Add temporal audio attention | CelebV-HQ test · Sync-C | 2.610 -> 2.689 | Helped | Sonic: Shifting Focus to Global Audio Perception in Portrait Animation (CVPR 2025 · 12 citations) |
| Add the discrete style codebook instead of regressive style extraction | MEAD · FID | 84.149 -> 59.718 | Helped | Say Anything with Any Style (AAAI 2024 · 11 citations) |
| Remove AU annotations during training | MEAD out-of-domain ablation evaluation · AU_F1 | 0.552 -> 0.435 | Hurt | InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generation (AAAI 2025 · 9 citations) |
| remove appearance loss | HDTF same-identity · LPIPS | 0.1565 -> 0.1735 | Hurt | FG-EmoTalk: Talking Head Video Generation with Fine-Grained Controllable Facial Expressions (AAAI 2024 · 9 citations) |
| Add V-AID decoupling instead of directly injecting the original audio embedding | HDTF · FID | 16.98 -> 16.64 | Helped | EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · 7 citations) |
| Add single-scale motion codebook compensation | VoxCeleb1 same-identity reconstruction · FID | 47.83 -> 49.00 | Hurt | Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation (CVPR 2025 · 5 citations) |
| use video labels 0--4 and 20--24, with audio labels 2 and 22, instead of video labels 0--2 and 2--4, with audio labels 1 and 3 | MTHM · Sync-C | 7.47 -> 7.56 | No clear effect | Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (NeurIPS 2025 · 5 citations) |
| Replace Whisper with funcodec as the audio condition encoder | HDTF · Sync-C | 7.696 -> 4.286 | Hurt | Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generation (CVPR 2025 · 4 citations) |
| Add tri-plane inversion | SD-Hybrid adaptation · CSIM | 0.823 -> 0.837 | Helped | MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes (NeurIPS 2024 · 4 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| Hallo | HDTF | FID | 14.2 – 20.64 | 3 papers, e.g. ICLR 2025 and CVPR 2025 |
| Wav2Lip | HDTF | CPBD | 0.299 – 0.79 | 2 papers, e.g. AAAI 2023 and AAAI 2022 |
| SadTalker | HDTF | FID | 18.32 – 22.18 | 2 papers, e.g. AAAI 2024 and CVPR 2025 |
| Sonic | HDTF | Sync-C | 8.35 – 9.139 | 2 papers, e.g. NeurIPS 2025 and ICML 2025 |
What reviewers will expect
Datasets and baselines named in the 30 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- HDTF · 13 of 30 papers
- MEAD · 7 of 30 papers
- CelebV-HQ · 6 of 30 papers
- VoxCeleb · 5 of 30 papers
- VFHQ · 4 of 30 papers
- VoxCeleb1 · 3 of 30 papers
- VoxCeleb2 · 3 of 30 papers
- RealTalk · 3 of 30 papers
Baselines
- SadTalker · 11 of 30 papers
- EchoMimic · 9 of 30 papers
- Hallo · 8 of 30 papers
- AniPortrait · 7 of 30 papers
- DreamTalk · 5 of 30 papers
- FOMM · 4 of 30 papers
- Wav2Lip · 4 of 30 papers
- Audio2Head · 4 of 30 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 66 papers; from OpenAlex and the papers' affiliations).
- Zhejiang University · 6 papers
- Korea Advanced Institute of Science and Technology · 4 papers
- Sun Yat-sen University · 4 papers
- Tsinghua University · 3 papers
- Shanghai Jiao Tong University · 3 papers
- Alibaba Group · 3 papers
- Peking University · 3 papers
- Tencent · 3 papers
- Microsoft · 3 papers
- Hong Kong University of Science and Technology · 3 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Diffusion-based human image animation (126 papers)
- Diffusion-based video editing (92 papers)
- Video-to-audio synthesis (76 papers)
- Text-to-motion diffusion (140 papers)
Elsewhere in the topic
- Efficient video diffusion models (283 papers)
- Trajectory-guided motion control (184 papers)
- Compositional text-to-video diffusion (159 papers)
- Autoregressive long video generation (156 papers)
- World-model video generation (110 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).