Research map / 3D vision / Face and shape
Speech-driven 3D facial animation
Synthesizes expressive 3D facial motion from audio using discrete priors and probabilistic generation.
32 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Face and shape, keyword cluster “3d face · face reconstruction · faces”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
11 papers in 2023–24 and 11 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 3.0% to 3.0%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Augmented Blendshapes for Real-Time Simultaneous 3D Head Modeling and Facial Motion Capture (CVPR 2016 · 27 citations)
- 2018 · Alive Caricature From 2D to 3D (CVPR 2018 · 55 citations)
- 2019 · Capture, Learning, and Synthesis of 3D Speaking Styles (CVPR 2019 · 366 citations)
- 2021 · Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (CVPR 2021 · 358 citations)
- 2022 · EMOCA: Emotion Driven Monocular Face Capture and Animation (CVPR 2022 · 221 citations)
- 2023 · SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation (CVPR 2023 · 312 citations)
- 2024 · SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis (CVPR 2024 · 76 citations)
- 2025 · DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · 11 citations)
- 2026 · Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation (NeurIPS 2026 · 0 citations)
Earliest papers here
- Augmented Blendshapes for Real-Time Simultaneous 3D Head Modeling and Facial Motion Capture (CVPR 2016 · 27 citations)
- Alive Caricature From 2D to 3D (CVPR 2018 · 55 citations)
- Capture, Learning, and Synthesis of 3D Speaking Styles (CVPR 2019 · 366 citations)
What has been tried
29 papers here report 255 measured changes (a component added, removed or swapped, compared with a baseline): helped 77%, no clear effect 5%, hurt 15%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Replace the Audio-Visual Encoder with HuBERT | May · LMD | 2.8032 -> 3.3961 | Hurt | SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis (CVPR 2024 · 76 citations) |
| Add the content encoder during training | 3D-HDTF-Test-A · FVE (×10^-6 mm) | 0.664 -> 0.551 | Helped | Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation (AAAI 2024 · 17 citations) |
| Replace autoregressive transformer with a convolutional architecture | Speech-driven facial motion synthesis · Sync Score | 0.4634 -> 0.217 | Hurt | Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks Methods and Applications (CVPR 2024 · 16 citations) |
| Replace TH-VQVAE with VAE | MEAD · Emo-FID | 15.15 -> 102.2 | Hurt | DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · 11 citations) |
| Replace the proposed representation with 3D SyncNet | VOCASET test · PLRS | 0.463 -> 0.435 | Hurt | Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics (CVPR 2025 · 5 citations) |
| Use joint motion-texture generation instead of motion-only generation | TexTalker4D-Test-A · LVE | 1.73 -> 1.49 | Helped | Towards High-fidelity 3D Talking Avatar with Personalized Dynamic Texture (CVPR 2025 · 3 citations) |
| add face-body motion decomposition (separate face and body motion estimation) | Talking-UB · L1 | 0.068 -> 0.035 | Helped | PTUS: Photo-Realistic Talking Upper-Body Synthesis via 3D-Aware Motion Decomposition Warping (AAAI 2024 · 3 citations) |
| Use ArcFace features collected from five images rather than identity conditioning from a single in-the-wild picture | 3D head identity conditioning · identity similarity | Helped | ID-to-3D: Expressive ID-guided 3D Heads via Score Distillation Sampling (NeurIPS 2024 · 2 citations) | |
| add 3DMM-based coarse control | HDTF · LMD | 2.187 -> 1.511 | Helped | Hierarchically Controlled Deformable 3D Gaussians for Talking Head Synthesis (AAAI 2025 · 1 citation) |
| add F0 conditioning | 3DTH-Test · Beat Alignment | 0.240 -> 0.306 | Helped | ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · 1 citation) |
| Add background music input | ChorusHead · PPE | 0.1295 -> 0.0948 | Helped | Let's Chorus: Partner-aware Hybrid Song-Driven 3D Head Animation (CVPR 2025 · 1 citation) |
| Add a single-scale appearance token T' | LRS3 · LPIPS | 0.109 -> 0.095 | Helped | TEASER: Token Enhanced Spatial Modeling for Expressions Reconstruction (ICLR 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 9 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- HDTF · 4 of 9 papers
- VOCASET · 4 of 9 papers
- MEAD · 3 of 9 papers
- BIWI · 3 of 9 papers
- LRS3 · 2 of 9 papers
Baselines
- FaceFormer · 6 of 9 papers
- CodeTalker · 6 of 9 papers
- SelfTalk · 4 of 9 papers
- DiffPoseTalk · 3 of 9 papers
- FaceDiffuser · 2 of 9 papers
- DECA · 2 of 9 papers
- SMIRK · 2 of 9 papers
- Deep3DFace · 2 of 9 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 22 papers; from OpenAlex and the papers' affiliations).
- Sun Yat-sen University · 3 papers
- Tencent · 2 papers
- South China University of Technology · 2 papers
- Guangdong University of Technology · 2 papers
- Cardiff University · 2 papers
Use it
Download BibTeX of 31 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- 3D morphable face model regression (80 papers)
- Face reflectance and relightable rendering (85 papers)
Elsewhere in the topic
- Neural implicit surface reconstruction (131 papers)
- Gaussian and implicit head avatars (115 papers)
- Textured mesh generation (111 papers)
- Primitive-based shape abstraction (99 papers)
- Single-view 3D reconstruction (96 papers)
- Full-body avatar reconstruction (78 papers)
- Articulated and non-rigid shape learning (75 papers)
- 3D shape generative modeling (74 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).