Research map / Video / Video generation
Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Playmate is a two-stage audio-driven portrait animation framework that trains a diffusion transformer over a decoupled implicit 3D (LivePortrait-style) motion space, then adds an emotion-control module for fine-grained emotion control.
Problem
Existing diffusion-based talking face methods couple lip-sync, head pose, and expression to audio, limiting independent control over head pose and emotional expression.
Approach
Generate motion sequences in a decoupled implicit 3D keypoint space (from LivePortrait) with a diffusion transformer conditioned on audio, using a motion-decoupled module (pairwise pose/expression transfer loss plus adaptive normalization with separate statistics for expression and pose), then add a frozen-backbone emotion-control module of two DiT blocks to inject emotion conditions.
Key results
- On HDTF, Playmate reaches FID 19.138 and FVD 231.048, versus 27.713 and 288.385 for the best baselines (MEMO and Hallo2).
- On the MEAD emotion test, Playmate reaches Emo-A 0.550 versus 0.460 for the next best (EDTalk), with FID 68.234 and FVD 149.837.
- In a 10-person MOS user study, Playmate scores highest on Video Definition (3.857) and Visual Appeal (3.643), while Sonic leads on Lip Sync (3.821 vs 3.750).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Playmate instead of MEMO | Portrait animation · HDTF | FID: 27.713 -> 19.138 | Helped |
| Use Playmate instead of Sonic | Portrait animation · HDTF | Sync-C: 9.139 -> 8.580 | Hurt |
| Use Playmate instead of Sonic | Portrait animation · Collected dataset | FID: 39.069 -> 34.716 | Helped |
| Use Playmate instead of Sonic | Portrait animation · Collected dataset | Sync-C: 7.972 -> 7.125 | Hurt |
| Use Playmate instead of EAT | Emotion-controlled portrait animation · MEAD test | Emo-A: 0.450 -> 0.550 | Helped |
| Add adaptive normalization for expression and head pose (ablation) | Playmate motion-decoupled module · Talking-face generation | Temporal continuity | Helped |
| Add Gaussian noise augmentation to previous motion representations (ablation) | Playmate diffusion transformer · Incremental video generation | Generated-image quality | Helped |
| Set audio CFG scale to 1.5 rather than 1.0 (ablation) | Playmate · CFG-scale ablation | Sync-C: 7.937 -> 8.141 | Helped |
| Set audio CFG scale to 2.0 rather than 1.5 (ablation) | Playmate · CFG-scale ablation | FID: 17.096 -> 15.968 | Helped |
| Set emotion CFG scale to 2.0 rather than 1.5 (ablation) | Playmate with audio CFG scale 1.5 · CFG-scale ablation | Emo-A: 54.405 -> 57.579 | Helped |
| Set emotion CFG scale to 2.0 rather than 1.5 (ablation) | Playmate with audio CFG scale 1.5 · CFG-scale ablation | FVD: 138.511 -> 173.114 | Hurt |
| Set emotion CFG scale to 2.5 rather than 1.5 (ablation) | Playmate with audio CFG scale 1.5 · CFG-scale ablation | FID: 18.948 -> 17.498 | Helped |
Surprising findings
- Sonic, a purely audio-driven method, achieves the best Sync-C and Sync-D on HDTF (9.139 and 6.549), beating Playmate on lip-sync metrics.
Limitations
- Processes primarily the face area; does not handle upper or full body.
- Lack of explicit 3D face model causes blurred edges during extreme head movements and texture sticking from neural rendering.
- Minor inconsistencies in complex backgrounds and limited robustness to diverse perspectives and styles.
Experimental setup
Datasets: AVSpeech CelebV-Text Acappella MEAD MAFW HDTF self-collected talking video dataset
Benchmarks: HDTF self-collected out-of-distribution test set MEAD test set
Metrics: FID FVD Sync-C Sync-D CSIM LPIPS Emo-A APD
Baselines: Hallo Hallo2 MEMO Sonic JoyVASA EAMM DreamTalk EDTalk EAT
Builds on: LivePortrait face-vid2vid VASA-1 DDPM
Closest papers by meaning
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.89)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.89)
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.88)
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · similarity 0.88)
- Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Control (ICLR 2025 · similarity 0.88)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.