Research map / 3D vision / Face and shape
DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DEEPTalk, a speech-driven 3D talking-head generator that uses a probabilistic joint speech-facial emotion embedding (DEE) and a temporally hierarchical VQ-VAE motion prior (TH-VQVAE) to produce diverse, emotionally expressive FLAME animations with accurate lip sync.
Problem
Speech-driven 3D facial animation produces blunt, monotonous expressions and fails to capture emotional nuance carried by speech, while lip sync and diversity are also hard to balance.
Approach
Learn a probabilistic cross-modal emotion embedding space from speech and FLAME expression sequences via contrastive learning, use it as an emotion prior and as an emotion-consistency loss, and predict hierarchical discrete motion codebook indices non-autoregressively so that sampling yields diverse, emotion-aligned motion.
Key results
- On MEAD, DEEPTalk achieves FID 26.07, FFD 64.02, Emo-FID 15.16 versus best baseline EMOTE Emo-FID 16.83 and FaceFormer FFD 78.62.
- On CREMA-D, FID 11.58 vs 14.78 for FaceDiffuser (second best); Emo-FID 11.99 vs 23.24.
- Diversity of 25.5 versus FaceDiffuser 21.56 while LSE-D stays at 10.60 vs 10.93 for FaceDiffuser.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace TH-VQVAE with VAE (ablation) | DEEPTalk · MEAD | Emo-FID: 15.15 -> 102.2 | Hurt |
| Replace TH-VQVAE with VQ-VAE (ablation) | DEEPTalk · MEAD | Emo-FID: 15.15 -> 16.58 | Hurt |
| Remove lip reading loss (ablation) | DEEPTalk · MEAD | LSE-D: 10.65 -> 10.91 | Hurt |
| Remove lip reading loss (ablation) | DEEPTalk · MEAD | FID: 26.07 -> 28.23 | No clear effect |
| Add emotion consistency loss (ablation) | DEEPTalk · MEAD | Emo-FID: 28.78 -> 15.15 | Helped |
| Add emotion consistency loss (ablation) | DEEPTalk · MEAD | LSE-D: 10.66 -> 10.65 | Hurt |
| Reduce emotion uncertainty with alpha=-4 instead of alpha=1 | DEEPTalk · MEAD test set | Diversity: 19.23 -> 25.55 | Helped |
| Use top temperature 4.5 and bottom temperature 1 instead of top 1 and bottom 0.1 | DEEPTalk · MEAD test set | Diversity: 9.91 -> 23.22 | Helped |
Surprising findings
- Vertex-based baselines trained on the small VOCASET scan dataset struggle with lip sync, suggesting pseudo-3D data from reconstructions is preferable for emotional animation.
- Adding the emotion consistency loss slightly worsens lip sync (LSE-D 10.65 vs 10.67 without, LSE-C 1.103 vs 1.107), indicating an emotion-lip sync tradeoff.
Limitations
- Training relies on pseudo-3D FLAME labels from 3D reconstruction, which are noisy for in-the-wild data and thus restricted to lab-recorded datasets.
- Emotion consistency loss creates a trade-off with lip synchronization accuracy.
- Evaluation of diversity and emotion relies partly on proxy metrics (FID-style, SyncNet) and a limited user study (98 participants, small sample of clips).
Experimental setup
Datasets: MEAD CREMA-D RAVDESS HDTF VOCASET AffectNet BEATv2 VoxCeleb2
Benchmarks: MEAD CREMA-D RAVDESS HDTF Emo-Vox
Metrics: FID FFD Emo-FID LSE-D LSE-C Diversity
Baselines: FaceFormer EMOTE FaceDiffuser EmoTalk MeshTalk CodeTalker
Builds on: VQ-VAE-2 PCME++ CodeTalker emotion2vec Gumbel-Softmax
Closest papers by meaning
- Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks Methods and Applications (CVPR 2024 · similarity 0.91)
- FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization (CVPR 2024 · similarity 0.90)
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.90)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.90)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.89)
- Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Model (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.