Research map / Video / Video generation
EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EmotiveTalk, a video-diffusion talking head framework that decouples audio into lip and expression latents (V-AID with Di-CTE) and injects them via an Expression Decoupling Injector for expressive, emotion-controllable, long-duration generation.
Problem
Audio-driven diffusion talking head generation struggles with expressiveness, controllability of emotions, high-resolution training cost, and stability over long videos due to coupled audio information and autoregressive inference.
Approach
Vision-guided decoupling of audio into lip-related and expression-related latents (via contrastive/MSE losses to vision latents plus a CLUB mutual-information constraint), a diffusion-based expression generator (Di-CTE), and an Expression Decoupling Injector that subtracts reference-portrait expression attention from driven-expression attention, combined with non-autoregressive overlapping-window
Key results
- On HDTF audio-only, Ours achieves FVD 140.96 and Sync-C 8.24 with E-FID 0.54, outperforming Hallo (FVD 276.31, Sync-C 7.99) and others.
- In the audio-video setting on HDTF, Ours reaches FID 16.09, FVD 120.70, E-FID 0.34, beating DreamTalk (FID 29.37, E-FID 0.55).
- User study (26 participants, 1-5 scale) gives Ours Lip-Sync 4.15 and Exp-Q 3.96, above DreamTalk (3.69, 3.45).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add V-AID decoupling instead of directly injecting the original audio embedding (ablation) | EmotiveTalk · HDTF | FID: 16.98 -> 16.64 | Helped |
| Add V-AID decoupling instead of directly injecting the original audio embedding (ablation) | EmotiveTalk · HDTF | Sync-C: 7.72 -> 8.24 | Helped |
| Add V-AID decoupling instead of directly injecting the original audio embedding (ablation) | EmotiveTalk · HDTF | E-FID: 0.66 -> 0.54 | Helped |
| Add lip-related latents (ablation) | EmotiveTalk · HDTF | FID: 16.02 -> 16.64 | Hurt |
| Add lip-related latents (ablation) | EmotiveTalk · HDTF | Sync-C: 0.65 -> 8.24 | Helped |
| Add lip-related latents (ablation) | EmotiveTalk · HDTF | E-FID: 0.57 -> 0.54 | Helped |
| Add expression-driven latents (ablation) | EmotiveTalk · HDTF | FID: 14.86 -> 16.64 | Hurt |
| Add expression-driven latents (ablation) | EmotiveTalk · HDTF | Sync-C: 8.04 -> 8.24 | Helped |
| Add expression-driven latents (ablation) | EmotiveTalk · HDTF | E-FID: 1.23 -> 0.54 | Helped |
| Use Di-CTE to expand utterance-driven emotion into frame-level expression latents (ablation) | EmotiveTalk · Expression generation | temporal expression variation | Helped |
Surprising findings
- Removing the expression-driven latents yields the best FID (14.86) on HDTF, which the authors attribute to higher similarity to the reference image.
Limitations
- Long-video stability evidence is shown only on audio-only HDTF with metrics that show small fluctuations, not a full perceptual study.
- Evaluation relies on open-source baseline reimplementations and a 10-sample user study with 26 participants.
Experimental setup
Datasets: HDTF MEAD
Benchmarks: HDTF MEAD
Metrics: FID FVD Sync-C Sync-D E-FID user study ratings
Baselines: SadTalker AniTalker AniPortrait Hallo PD-FGC StyleTalk DreamTalk
Builds on: Hallo Stable Video Diffusion MimicMotion CLUB DDIM
Closest papers by meaning
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.92)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.90)
- Diffusion-based Realistic Listening Head Generation via Hybrid Motion Modeling (CVPR 2025 · similarity 0.90)
- FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization (CVPR 2024 · similarity 0.90)
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · similarity 0.89)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.