Research map / Video / Video generation
InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InstructAvatar, a two-branch diffusion motion generator that drives 2D talking avatars from audio plus natural-language emotion and facial-motion instructions, trained on an automatically annotated instruction-video dataset.
Problem
Talking avatar generation achieves good lip-sync but offers limited, coarse control over detailed facial expressions and emotions; prior emotion control relies on labels, reference videos, or 3D animation pipelines.
Approach
Converts emotion labels and action units (AUs, paraphrased by GPT-4V) into fine-grained text instructions, and injects them via two cross-attention branches (EOS token for emotion, all-token states for motion) into a Conformer diffusion motion generator, gated by zero-convolution to preserve a pretrained talking-head model.
Key results
- In-domain on MEAD, InstructAvatar achieves AU F1 0.738 and FID 44.593, the best among compared methods, with Sync_D 9.412 (second-best).
- Subjective MOS on in-domain MEAD: lip-sync 4.74, naturalness 4.63, jitter 4.88, outperforming EAT, StyleTalk and DreamTalk.
- Removing the zero-convolution gate raises Sync_D from 9.653 to 12.832, showing it is critical for lip-sync.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use text-guided InstructAvatar rather than the strongest listed baseline | Emotional talking generation on MEAD in-domain · MEAD in-domain | AU_F1: 0.711 -> 0.738 | Helped |
| Use text-guided InstructAvatar rather than the strongest listed baseline | Emotional talking generation out-of-domain · TalkingHead 1KH | AU_F1: 0.542 -> 0.552 | Helped |
| Use text-guided InstructAvatar rather than the strongest listed baseline | Emotional talking generation on MEAD in-domain · MEAD in-domain | AU_Emo: 0.593 -> 0.566 | Hurt |
| Use text-guided InstructAvatar rather than the strongest listed baseline | Emotional talking generation on MEAD in-domain · MEAD in-domain | FID: 47.269 -> 44.593 | Helped |
| Remove AU annotations during training (ablation) | InstructAvatar · MEAD out-of-domain ablation evaluation | AU_F1: 0.552 -> 0.435 | Hurt |
| Combine emotion and motion learning into one branch (ablation) | InstructAvatar · MEAD out-of-domain ablation evaluation | AU_F1: 0.552 -> 0.488 | Hurt |
| Remove zero convolution (ablation) | InstructAvatar · MEAD out-of-domain ablation evaluation | Sync_D: 9.653 -> 12.832 | Hurt |
| Substitute the key-frame latent with a frame from another emotion type during training (ablation) | InstructAvatar · TalkingHead 1KH out-of-domain | AU_F1: 0.339 -> 0.552 | Helped |
| Remove CLIP Adapter (ablation) | InstructAvatar · Ablation evaluation | AU_F1: 0.537 -> 0.512 | Hurt |
| Remove AU loss (ablation) | InstructAvatar · Ablation evaluation | AU_F1: 0.537 -> 0.504 | Hurt |
| Replace pseudo-empty audio with all-zero tensors (ablation) | InstructAvatar · Ablation evaluation | Mot.: 4.46 -> 3.23 | Hurt |
| Use emotion labels instead of text guidance (ablation) | InstructAvatar · Ablation evaluation | AU_F1: 0.537 -> 0.469 | Hurt |
Surprising findings
- Text-only emotion control outperforms video-reference methods (StyleTalk, DreamTalk) on AU F1 despite using less information at inference.
- Using all-zero tensors instead of pseudo-empty audio for motion-only control sharply degrades motion control.
Limitations
- Trained only on AUs extracted from real talking videos, limiting disentangled single-AU control.
- Relatively modest training dataset may hinder robustness to highly out-of-domain instructions or appearances.
- Almost all training data follows a single emotion/motion pattern, so simultaneous emotion and motion control is difficult.
Experimental setup
Datasets: MEAD HDTF CC v1 TalkingHead 1KH
Benchmarks: MEAD TalkingHead 1KH CC v1
Metrics: AU F1 AU Emo FID Sync_D CLIP_S MOS (emotion, lip-sync, jitter, naturalness, motion accuracy
Baselines: GAIA MakeItTalk EAT StyleTalk DreamTalk
Builds on: GAIA Conformer DDPM DDIM CLIP
Closest papers by meaning
- MVPortrait: Text-Guided Motion and Emotion Control for Multi-view Vivid Portrait Animation (CVPR 2025 · similarity 0.87)
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.87)
- FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization (CVPR 2024 · similarity 0.87)
- MoEE: Mixture of Emotion Experts for Audio-Driven Portrait Animation (CVPR 2025 · similarity 0.87)
- GAIA: Zero-shot Talking Avatar Generation (ICLR 2024 · similarity 0.87)
- Towards High-fidelity 3D Talking Avatar with Personalized Dynamic Texture (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.