Research map / Image generation / Editing and style transfer
MoEE: Mixture of Emotion Experts for Audio-Driven Portrait Animation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoEE, a mixture-of-emotion-experts diffusion model for audio-driven portrait animation with multimodal emotion control, plus the 150-hour DH-FaceEmoVid-150 emotion dataset.
Problem
Audio-driven talking head generation lacks precise modeling of single basic emotions and compound emotions, and existing emotional talking-head datasets are small and have only tag-level labels.
Approach
Six per-emotion cross-attention experts inside a denoising U-Net, combined via soft global and local gating so compound emotions blend experts; an emotion-to-latents module maps text, audio, and label conditions into a shared emotion latent via attention; training uses masked noisy emotion sampling and a timestep-aware spatial loss.
Key results
- On DH-FaceEmoVid-150, MoEE reaches FID 39.619 and FVD 402.803 versus best baseline EAT at FID 48.011 and FVD 467.739.
- On HDTF, MoEE achieves FVD 322.625 and LPIPS 0.152, better than Hallo (FVD 343.023), though Sync-C (6.114) and Sync-D (9.101) are slightly weaker than some baselines.
- Ablation removing the DH-FaceEmoVid-150 dataset raises FVD from 402.803 to 511.928 and LPIPS from 0.182 to 0.275.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Mixture of Emotion Experts (ablation) | MoEE · DH-FaceEmoVid-150 | FID: 58.411 -> 39.619 | Helped |
| add Global Soft Assignment (ablation) | MoEE · DH-FaceEmoVid-150 | FID: 46.334 -> 39.619 | Helped |
| add Masked Noisy Emotion Sample strategy (ablation) | MoEE · DH-FaceEmoVid-150 | FID: 51.788 -> 39.619 | Helped |
| train with DH-FaceEmoVid-150 dataset (ablation) | MoEE · DH-FaceEmoVid-150 | FID: 52.412 -> 39.619 | Helped |
Surprising findings
- Sync-C and Sync-D slightly decline on HDTF, which the authors attribute to the Chinese talking-head dataset used.
Limitations
- Compound emotions are poorly represented due to limited compound categories; e.g., 'sadly disgusted' emphasizes disgust rather than blending.
- Fine-grained AU control is trained only on AU combinations from real videos, limiting disentangled single-AU control.
- Only six basic and four compound emotions covered.
Experimental setup
Datasets: HDTF DH-FaceVid-1K MEAD DH-FaceEmoVid-150
Benchmarks: HDTF MEAD DH-FaceEmoVid-150
Metrics: FID FVD LPIPS Sync-C Sync-D AED AKD
Baselines: AniPortrait Hallo EchoMimic StyleTalk EAT
Builds on: Soft MoE Stable Diffusion AnimateDiff-style temporal attention wav2vec GPT-4V annotation pipeline
Closest papers by meaning
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.88)
- FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization (CVPR 2024 · similarity 0.88)
- InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generation (AAAI 2025 · similarity 0.87)
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · similarity 0.87)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.87)
- EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.