Research map / 3D vision / Face and shape
ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProsodyTalker, which decomposes speech prosody into content (lip) and F0 (head pose) streams, and builds a 3D talking-head model with a content-oriented fusion decoder plus an F0-conditioned latent diffusion head-pose generator, along with a new 4D head-pose dataset (3DTH).
Problem
3D visual speech animation methods synchronize lips with speech but ignore head poses, and head pose synthesis suffers from an uncertain speech-to-pose mapping and a lack of 4D face datasets with realistic head poses.
Approach
Explicitly split speech into language content (driving lip movements via information perturbation and a content encoder) and F0 (driving head poses), then generate lips autoregressively with a content-motion fusion decoder and generate head poses with an F0-conditioned latent diffusion model over a transformer VAE latent space.
Key results
- On BIWI-Test-A, the method reaches LVE 3.8430 versus 4.2485 for SelfTalk (best baseline).
- On 3DTH-Test head pose, beat alignment 0.306 vs 0.295 for DiffPoseTalk, with diversity 1.182 vs 1.169.
- In the user study on VOCASET-Test, ours is preferred over SelfTalk for lip-sync 49.24% and realism 63.17%, and ours vs GT is 43.21% lip-sync and 47.24% realism.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add F0 conditioning (ablation) | ProsodyTalker head-pose synthesis · 3DTH-Test | Beat Alignment: 0.240 -> 0.306 | Helped |
| add F0 conditioning (ablation) | ProsodyTalker head-pose synthesis · 3DTH-Test | Diversity: 0.427 -> 1.182 | Helped |
| add F0 normalization (ablation) | ProsodyTalker head-pose synthesis · 3DTH-Test | Beat Alignment: 0.218 -> 0.306 | Helped |
| add F0 normalization (ablation) | ProsodyTalker head-pose synthesis · 3DTH-Test | Diversity: 1.144 -> 1.182 | Helped |
| replace the head-pose codebook with latent diffusion (ablation) | ProsodyTalker · 3DTH-Test | Diversity: 0.648 -> 1.182 | Helped |
| add information perturbation (ablation) | ProsodyTalker content encoder · VOCASET-Test | LVE: 5.9245 -> 2.8246 | Helped |
| add information perturbation (ablation) | ProsodyTalker content encoder · VOCASET-Test | FDD: 1.6257 -> 0.8649 | Helped |
| add pitch randomization (ablation) | ProsodyTalker content encoder · VOCASET-Test | LVE: 3.4445 -> 2.8246 | Helped |
| add pitch randomization (ablation) | ProsodyTalker content encoder · VOCASET-Test | FDD: 0.9525 -> 0.8649 | Helped |
| add formant shifting (ablation) | ProsodyTalker content encoder · VOCASET-Test | LVE: 7.5441 -> 2.8246 | Helped |
| add formant shifting (ablation) | ProsodyTalker content encoder · VOCASET-Test | FDD: 1.9875 -> 0.8649 | Helped |
Surprising findings
- The method approximates ground-truth animation realism in the user study (ours vs GT near 50% preference).
Limitations
- 3DTH dataset is reconstructed from 2D talking-head videos with EMOCA-v2, so it inherits reconstruction jitter and noise (mitigated by sliding window).
- Head pose evaluation is limited to a single beat-alignment and diversity metric on the authors' own dataset.
- Lip-sync evaluation excludes head poses since benchmark datasets are neutral-pose.
Experimental setup
Datasets: VOCASET BIWI MEAD HDTF 3DTH
Benchmarks: VOCASET-Test BIWI-Test-A 3DTH-Test
Metrics: LVE FDD beat alignment diversity user preference (lip-sync and realism)
Baselines: FaceFormer CodeTalker SelfTalk SadTalker DiffPoseTalk Wav2vec2 HuBERT
Builds on: NANSY FaceFormer MLD ALiBi EMOCA-v2
Closest papers by meaning
- Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks Methods and Applications (CVPR 2024 · similarity 0.92)
- Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation (NeurIPS 2026 · similarity 0.90)
- Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics (CVPR 2025 · similarity 0.89)
- Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation (AAAI 2024 · similarity 0.89)
- From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech (CVPR 2025 · similarity 0.89)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.