Research map / Video / Video generation
KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
KeyFace generates long audio-driven talking-face videos by first producing low-frame-rate keyframes conditioned on audio, identity and valence/arousal, then interpolating the gaps with a second diffusion model, and adds non-speech vocalization handling.
Problem
Audio-driven facial animation suffers from error accumulation and identity drift over long durations, and existing methods neglect continuous emotion and non-speech vocalizations (NSVs) such as laughter and sighs.
Approach
A two-stage keyframe-then-interpolation latent video diffusion framework built on Stable Video Diffusion: sparse keyframes carry audio-driven motion and identity via concatenation (no ReferenceNet or motion frames), an interpolation model fills the gaps, and valence/arousal embeddings give continuous emotion control; WavLM and BEATs audio features cover speech and NSVs.
Key results
- On HDTF, KeyFace achieves FID 16.76 and FVD 137.25, the lowest among compared methods (best Hallo FID 19.22, V-Express FVD 200.67).
- On MEAD, KeyFace with valence/arousal conditioning reaches Emoacc 0.67 vs 0.72 for EDTalk, and FVD 447.74 vs 560.61 for EAT with discrete labels.
- User study Elo on HDTF gives KeyFace 1091.52, the highest among methods (Hallo 1054.69, EchoMimic 1023.53).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use continuous valence and arousal instead of discrete emotion labels (ablation) | KeyFace · MEAD | Emoacc: 0.43 -> 0.67 | Helped |
| Use BEATs with WavLM instead of WavLM alone (ablation) | KeyFace audio encoder · HDTF | NSVacc: 0.10 -> 0.42 | Helped |
| Remove BEATs from the combined audio encoder (ablation) | KeyFace audio encoder · HDTF | NSVacc: 0.42 -> 0.10 | Hurt |
| Use WavLM instead of Wav2Vec2 with BEATs (ablation) | KeyFace audio encoder · HDTF | LipScore: 0.32 -> 0.36 | Helped |
| Remove temporal layers from keyframe generation (ablation) | KeyFace · HDTF | FID: 16.76 -> 23.74 | Hurt |
| Replace concatenation with ReferenceNet (ablation) | KeyFace · HDTF | FID: 16.76 -> 39.71 | Hurt |
| Add pixel-space loss (ablation) | KeyFace · HDTF | FID: 18.76 -> 19.00 | Helped |
| Use L1 rather than L2 pixel loss (ablation) | KeyFace · HDTF | FVD: 137.54 -> 172.01 | Hurt |
| Add perceptual loss to L2 pixel loss (ablation) | KeyFace · HDTF | FID: 19.00 -> 16.76 | Helped |
| Set lower-image pixel-loss weight to 3 instead of 1 (ablation) | KeyFace · HDTF | FVD: 186.87 -> 137.25 | Helped |
| Train on high-quality data only rather than all training data (ablation) | KeyFace keyframe model · HDTF | FID: 26.92 -> 16.97 | Helped |
| Train the interpolation model on high-quality data only rather than all training data (ablation) | KeyFace interpolation model · HDTF | LipScore: 0.24 -> 0.36 | Helped |
Surprising findings
- Training exclusively on high-quality data improves lip sync for the interpolation model and video quality for the keyframe model, but mixing in lower-quality CelebV data hurt performance.
- Using classifier-free guidance on both stages makes videos overly static, while autoguidance on the keyframe model lowers LipScore.
Limitations
- Relies on a pre-trained emotion recognition model to produce pseudo-labels for valence and arousal training.
- Lip-sync and NSV metrics are newly introduced and rely on a custom lipreader and classifier, limiting comparability.
- Hallo and other baselines are reported on the same HDTF test set but the main training includes in-house data not released.
Experimental setup
Datasets: HDTF CelebV-Text CelebV-HQ MEAD
Benchmarks: HDTF CelebV-Text MEAD
Metrics: AQ FID LPIPS FVD smoothness LipScore Emoacc NSVacc Elo
Baselines: SadTalker Hallo V-Express AniPortrait EchoMimic EDTalk EAT EAMM
Builds on: Stable Video Diffusion EDM classifier-free guidance autoguidance
Closest papers by meaning
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.90)
- Sonic: Shifting Focus to Global Audio Perception in Portrait Animation (CVPR 2025 · similarity 0.90)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.89)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.89)
- Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Model (ICML 2025 · similarity 0.89)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.