Research map / Vision-language and multimodal / Cross-modal learning
EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds on MegaPortraits with latent-space and loss changes (canonical volume loss, source-driver mismatch loss, 128-dim expression latent, PCA mouth loss) for intense expression transfer and audio-driven animation, and releases the multi-view FEED dataset of extreme expressions.
Problem
One-shot head avatars based on MegaPortraits fail to transfer intense, asymmetric facial expressions in cross-driving synthesis, and existing expression datasets lack extreme and asymmetric multi-view face videos.
Approach
Analyze MegaPortraits' expression latent space via PCA (AUC of explained variance), remove expression leakage into the canonical volume with a matching loss, add a self-supervised source-driver mismatch loss and a 128-dim latent, and disentangle head pose from expression so the expression encoder can be taught by an audio encoder using PCA-isolated mouth components.
Key results
- Image-driven on MEAD: FID 59.6 and CSIM 0.74 vs MegaPortraits 61.1 and 0.73; user preference for motion (UMTN) 70.3% vs 22.7%.
- Speech-driven on HDTF: FID 28.5 vs SadTalker 55.6 and StyleHEAT 62.4, with best sync offset 0.07.
- Ablation: fine-tuning MegaPortraits on FEED did not improve results (UMTN 8.5 vs 45.4 for the full model).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EMOPortraits instead of MegaPortraits | Image-driven cross-reenactment · MEAD high-intensity expressions | UMTN: 22.7 -> 70.3 | Helped |
| Use EMOPortraits instead of MegaPortraits | Image-driven cross-reenactment · FEED extreme expressions | UMTN: 16.7 -> 74.6 | Helped |
| Use EMOPortraits instead of MegaPortraits | Image-driven cross-reenactment · FEED extreme expressions | CSIM: 0.73 -> 0.70 | Hurt |
| Use EMOPortraits instead of MegaPortraits | Image-driven cross-reenactment · MEAD high-intensity expressions | FID: 61.1 -> 59.6 | Helped |
| Use EMOPortraits instead of MegaPortraits | Image-driven cross-reenactment · FEED extreme expressions | FID: 62.8 -> 60.2 | Helped |
| Use EMOPortraits instead of leading speech-driven baselines | Speech-driven animation · HDTF | MP: 2.53 vs 2.56 | Mixed |
| Remove canonical volume loss L_n^CV (ablation) | EMOPortraits image-driven mode · MEAD-driven cross-reenactment | UMTN: 45.4 -> 9.2 | Hurt |
| Remove source-driver mismatch loss L_sdm (ablation) | EMOPortraits image-driven mode · MEAD-driven cross-reenactment | CSIM: 0.74 -> 0.59 | Hurt |
| Set expression dimension dim(z) to 512 instead of 128 (ablation) | EMOPortraits image-driven mode · MEAD-driven cross-reenactment | CSIM: 0.74 -> 0.44 | Hurt |
| Fine-tune MegaPortraits on FEED (ablation) | MegaPortraits · FEED-driven cross-reenactment | UMTN: 8.5 | No clear effect |
| Remove head-pose and expression latent-space disentanglement (ablation) | EMOPortraits speech-driven mode · HDTF | FID: 28.5 -> 44.7 | Hurt |
| Remove mouth PCA loss L_PCA (ablation) | EMOPortraits speech-driven mode · HDTF | MP: 0.82 -> 0.84 | Hurt |
Surprising findings
- Fine-tuning the pretrained MegaPortraits on the FEED dataset did not improve emotion transfer and reduced identity preservation due to overfitting.
- Ground-truth videos scored worse than the method on SyncNet offset, suggesting SyncNet metrics may not fully reflect sync quality.
Limitations
- Does not generate the avatar's body or shoulders; requires a source image body.
- Sometimes struggles with accurate expression translation.
- Underperforms with extensive head rotation.
Experimental setup
Datasets: VoxCeleb2 FEED MEAD FFHQ HDTF
Benchmarks: MEAD FEED HDTF
Metrics: FID CSIM UMTN UAPP MAE landmark position and velocity SyncNet offset and confidence AUCz
Baselines: MegaPortraits FOMM UVA StyleHEAT MetaPortrait NOFA SadTalker MakeItTalk PC-AVS
Builds on: MegaPortraits CosFace FOMM SadTalker
Closest papers by meaning
- RealPortrait: Realistic Portrait Animation with Diffusion Transformers (AAAI 2025 · similarity 0.87)
- MVPortrait: Text-Guided Motion and Emotion Control for Multi-view Vivid Portrait Animation (CVPR 2025 · similarity 0.87)
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · similarity 0.87)
- MoEE: Mixture of Emotion Experts for Audio-Driven Portrait Animation (CVPR 2025 · similarity 0.87)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.87)
- X-NeMo: Expressive Neural Motion Reenactment via Disentangled Latent Attention (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.