Research map / Video / Video generation
Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Cafe-Talk is a diffusion-transformer 3D talking face model that adds coarse (style, emotion, intensity) and fine-grained AU-based spatiotemporal control via a two-stage training pipeline, swap-label training, mask-based CFG, and a text-to-AU detector.
Problem
Speech-driven 3D talking face generation lacks flexible fine-grained, spatiotemporal facial expression control; existing methods use only global emotion labels or AU refinements of emotion, and fine-grained conditions entangle with speech and hurt lip sync.
Approach
A coarse-to-fine two-stage pipeline: a frozen-after-stage-1 diffusion transformer conditioned on audio and coarse labels, plus a zero-convolution fine-grained AU adapter trained with swapped emotion labels so AU conditions dominate in masked regions; inference uses a temporally masked CFG, and a CLIP-adapter text detector maps natural language to AUs.
Key results
- On MEAD, Cafe-Talk reports LVE 7.21, emotion Acc 59.48% and Div 119.36, beating EmoTalk (7.90, 9.97%, 62.02) and UniTalker.
- On LibriSpeech, SyncC 5.55 vs 3.30 for EmoTalk and 4.72 for UniTalker, though SyncD 9.76 is slightly worse than UniTalker's 9.70.
- Ablation on fine-grained control: removing swap-label drops Control Rate from 0.51 to 0.12, and removing the two-stage pipeline raises SyncD to 11.94 from 10.14.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AdaLN self-attention instead of FiLM cross-attention for coarse-grained conditioning (ablation) | Cafe-Talk base model · Coarse-grained design ablation | emotion accuracy: 48.19\% -> 59.48\% | Helped |
| use AdaLN self-attention instead of FiLM cross-attention for coarse-grained conditioning (ablation) | Cafe-Talk base model · Coarse-grained design ablation | LVE: 7.64 -> 7.21 | Helped |
| add the two-stage pipeline rather than jointly train coarse- and fine-grained control (ablation) | Cafe-Talk · Fine-grained control test | Control Rate: 0.35 -> 0.51 | Helped |
| add the two-stage pipeline rather than jointly train coarse- and fine-grained control (ablation) | Cafe-Talk · Fine-grained control test | SyncD: 11.94 -> 10.14 | Helped |
| add swap-label training (ablation) | Cafe-Talk fine-grained adapter · Fine-grained control test | Control Rate: 0.12 -> 0.51 | Helped |
| add swap-label training (ablation) | Cafe-Talk fine-grained adapter · Fine-grained control test | SyncC: 4.07 -> 5.38 | Helped |
| use mask-based CFG instead of the alternative CFG technique (ablation) | Cafe-Talk inference · Fine-grained control test | Control Rate: 0.27 -> 0.51 | Helped |
| use mask-based CFG instead of the alternative CFG technique (ablation) | Cafe-Talk inference · Fine-grained control test | SyncC: 4.74 -> 5.38 | Helped |
| add emotion intensity conditioning (ablation) | Cafe-Talk base model · MEAD test | emotion accuracy: 53.27\% -> 59.48\% | Helped |
| add emotion intensity conditioning (ablation) | Cafe-Talk base model · MEAD test | expression diversity: 115.17 -> 119.36 | Helped |
| add InfoNCE alignment loss (ablation) | text-based AU detector · Textual AU detection, in-domain template | F1-score: 0.98 -> 0.92 | Hurt |
| add fine-grained AU control to the model | Cafe-Talk · Single-AU fine-grained control | SyncD: +0.156 on average; +0.693 | Mixed |
Surprising findings
- Lower-face AU conditions conflicting with speech substantially degrade lip synchronization (SyncD +0.693 on average vs +0.156 for upper-face AUs).
- Swapping emotion labels to conflict with audio lowers emotion accuracy to 10.86% when the audio is taken as ground truth, showing that the audio drives expression when labels are swapped.
Limitations
- Inaccurate lip movements when lower-face AU conditions explicitly conflict with the speech audio.
- Inference latency is high due to diffusion sampling (14.71 s for 5 s audio).
- Text-AU detector trained on a small GPT-generated set (228 pairs) plus AffectNet-derived pairs.
Experimental setup
Datasets: MEAD RAVDESS LibriSpeech AffectNet in-house internet multilingual speech video dataset (157 h)
Benchmarks: MEAD RAVDESS LibriSpeech
Metrics: LVE SyncD SyncC Acc Div Control Rate (CR) F1
Baselines: EmoTalk UniTalker FaceFormer TalkSHOW EMOTE
Builds on: DDPM FaceFormer CLIP-Adapter classifier-free guidance ControlNet zero convolution
Closest papers by meaning
- Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics (CVPR 2025 · similarity 0.89)
- Learning Adaptive Spatial Coherent Correlations for Speech-Preserving Facial Expression Manipulation (CVPR 2024 · similarity 0.88)
- Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion (ICML 2025 · similarity 0.88)
- KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation (CVPR 2025 · similarity 0.88)
- FG-EmoTalk: Talking Head Video Generation with Fine-Grained Controllable Facial Expressions (AAAI 2024 · similarity 0.88)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.