Research map / Video / Video generation
SPACE: Speech-driven Portrait Animation with Controllable Expression
Siddharth Gururani, Arun Mallya, Ting-Chun Wang, Rafael Valle, Ming-Yu Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-stage landmarks plus latent keypoints with face-vid2vid generator instead of baselines | SPACE · Held-out test set | FID: 15.67 -> 11.68 (vs Wav2Lip) | Improved |
| use SPACE instead of PC-AVS, MakeItTalk or Wav2Lip | SPACE · Human A/B preference | preference for SPACE: 89.1%, 87.8%, 72.6% preferred over PC-AVS, MakeItTalk, Wav2Lip | Improved |
| use SPACE instead of PC-AVS on lip sync | SPACE · Held-out test set | SyncNet confidence: 7.00 vs 3.61 | Hurt |
| add landmark intermediate stage (multi-stage) instead of directly predicting latent keypoints from speech (ablation) | SPACE · Held-out test set | FID: 11.82 -> 11.68 | No clear effect |
| add landmark intermediate stage (multi-stage) instead of directly predicting latent keypoints from speech (ablation) | SPACE · Held-out test set | mouth position error (M-P): 0.021 -> 0.025 | Hurt |
| give L2L audio input in addition to landmarks (ablation) | SPACE · Held-out test set | FID: 15.63 -> 11.68 | Improved |
| give L2L audio input in addition to landmarks (ablation) | SPACE · Held-out test set | SyncNet confidence: 2.78 -> 3.61 | Improved |
| train S2L on normalized (frontalized) landmarks instead of raw landmarks (ablation) | SPACE · Qualitative | mouth motion accuracy and pose drift | Improved |
Closest papers by meaning
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.88)
- Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion (ICML 2025 · similarity 0.87)
- Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer (CVPR 2025 · similarity 0.86)
- KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation (CVPR 2025 · similarity 0.86)
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · similarity 0.86)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
