Research map / Video / Video generation
VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VASA-1, which generates real-time lifelike talking faces from one portrait and audio by running a diffusion transformer over a disentangled 3D face latent space.
Problem
Audio-driven talking face generation that is lip-synced and also expressive, with natural head motion, while running fast enough for real-time interaction.
Approach
Model all facial dynamics (lip, expression, gaze, blink) and head pose jointly as one latent sequence with a diffusion transformer, on top of a 3D-aided face latent space trained with added disentanglement losses (pose-dynamics consistency and cross-identity face similarity).
Key results
- On VoxCeleb2, the method reaches S_C 8.841 and S_D 6.312 vs. SadTalker 5.843 and 8.813, and CAPP 0.468 vs. 0.441.
- On OneMin-32, FVD_25 is 105.88 vs. 209.77 for Audio2Head and 214.51 for SadTalker.
- Generates 512x512 video at up to 40 FPS online with 170 ms latency on a single RTX 4090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use holistic facial dynamics and head-pose generation in a disentangled face latent space | VASA-1 · VoxCeleb2 | CAPP: 0.441 -> 0.468 | Helped |
| Use holistic facial dynamics and head-pose generation in a disentangled face latent space | VASA-1 · OneMin-32 | FVD_25: 209.77 -> 105.88 | Helped |
| Train on 10% of the data instead of the full dataset (ablation) | VASA-1 · OneMin-32 | FVD_25: 105.88 -> 147.401 | Hurt |
| Train on 10% of the data instead of the full dataset (ablation) | VASA-1 · OneMin-32 | S_C: 7.957 -> 7.990 | No clear effect |
| Set audio CFG scale λ_A to 0.0 instead of 0.5 (ablation) | VASA-1 · OneMin-32 | S_C: 7.134 vs 7.957 | Hurt |
| Increase audio CFG scale λ_A from 0.5 to 1.0 (ablation) | VASA-1 · OneMin-32 | CAPP: 0.465 -> 0.474 | Mixed |
| Increase gaze CFG scale λ_g from 0.0 to 2.0 (ablation) | VASA-1 · OneMin-32 | E_θg: 5.730 -> 5.064 | Helped |
| Remove the pairwise head-pose and facial-dynamics transfer loss l_consist (ablation) | VASA-1 face latent model · Face latent disentanglement | facial dynamics preservation | Hurt |
| Add cross-identity face identity loss l_cross_id (ablation) | VASA-1 face latent model · VoxCeleb2 cross-reenactment | identity preservation cosine similarity: 0.72 -> 0.80 | Helped |
| Reduce diffusion sampling steps from 50 to 10 (ablation) | VASA-1 · OneMin-32 | audio-lip and audio-pose alignment | Helped |
| Reduce diffusion sampling steps from 50 to 10 (ablation) | VASA-1 · OneMin-32 | FVD_25 | Hurt |
| Increase audio CFG scale beyond 0.5 (ablation) | VASA-1 · OneMin-32 | audio-pose synchronization | Hurt |
Surprising findings
- Lip-audio sync scores with audio CFG exceed those of real videos (S_C 7.957 vs. 7.192 on OneMin-32).
- Training on only 10% of data (50K clips) gives comparable sync scores but worse FVD and pose intensity.
Limitations
- Processes human regions only up to the torso.
- Lacks an explicit 3D face model, which can cause texture-sticking artifacts.
- Does not model non-rigid elements such as hair and clothing.
Experimental setup
Datasets: VoxCeleb2 in-house high-resolution talking video dataset (~3.5K subjects) OneMin-32 (collected by authors, 32 one-minute clips of 17 individuals)
Benchmarks: VoxCeleb2 test subset (460 clips) OneMin-32
Metrics: SyncNet confidence S_C SyncNet distance S_D CAPP pose variation intensity FVD_25 identity similarity gaze angular error head distance error
Baselines: MakeItTalk Audio2Head SadTalker
Builds on: 3D-aided face reenactment framework Wav2Vec2 classifier-free guidance CLIP
Closest papers by meaning
- VASA-3D: Lifelike Audio-Driven Gaussian Head Avatars from a Single Image (NeurIPS 2025 · similarity 0.92)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.90)
- FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization (CVPR 2024 · similarity 0.90)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.89)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.89)
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.