Research map / Video / Video generation
FG-EmoTalk: Talking Head Video Generation with Fine-Grained Controllable Facial Expressions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FG-EmoTalk, a GAN-based one-shot talking head framework that disentangles appearance, pose and expression latents and edits expression via facial Action Unit intensities, trained self-supervised.
Problem
Fine-grained, controllable facial expression editing in one-shot talking head video generation, beyond coarse discrete emotion labels or copying expressions from driving videos.
Approach
Disentangles the face into appearance, pose and expression components (with a dedicated appearance extractor), and aligns an AU encoder (learnable AU embedding + Gated-GCN) with the expression latent so that AU intensity vectors can directly edit expression details; trained with self-supervised swapped-latent image generation and appearance/expression losses.
Key results
- Expression-quality accuracy (ACC-emo) of 0.6102 vs 0.5773 for VideoRetalking and 0.5435 for EAMM.
- Visual quality: LPIPS 0.1565, PSNR 31.47, SSIM 0.797 on same-identity setting, best among TPSM, StyleHEAT, LIA, DPE.
- User study with 20 participants: selected as best in 63.5% to 68% of cases across naturalness, appearance, lip sync, expression and overall.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove appearance loss (ablation) | FG-EmoTalk · HDTF same-identity | LPIPS: 0.1565 -> 0.1735 | Hurt |
| remove appearance loss (ablation) | FG-EmoTalk · HDTF same-identity | PSNR: 31.4725 -> 29.4573 | Hurt |
| remove appearance loss (ablation) | FG-EmoTalk · HDTF same-identity | SSIM: 0.7970 -> 0.7404 | Hurt |
| remove appearance loss (ablation) | FG-EmoTalk · HDTF same-identity | APD: 0.0067 -> 0.0186 | Hurt |
| remove appearance loss (ablation) | FG-EmoTalk · HDTF same-identity | AED: 0.0053 -> 0.0214 | Hurt |
| remove expression loss (ablation) | FG-EmoTalk · HDTF same-identity | LPIPS: 0.1565 -> 0.1690 | Hurt |
| remove expression loss (ablation) | FG-EmoTalk · HDTF same-identity | PSNR: 31.4725 -> 30.8425 | Hurt |
| remove expression loss (ablation) | FG-EmoTalk · HDTF same-identity | SSIM: 0.7970 -> 0.7574 | Hurt |
| remove expression loss (ablation) | FG-EmoTalk · HDTF same-identity | APD: 0.0067 -> 0.0307 | Hurt |
| remove expression loss (ablation) | FG-EmoTalk · HDTF same-identity | AED: 0.0053 -> 0.0117 | Helped |
| add AU intensity control for expression editing | FG-EmoTalk · MEAD | ACCemo: 0.5773 -> 0.6102 | Helped |
Limitations
- Only 7 AUs retained for control after filtering poorly labeled DISFA examples and mouth-affecting AUs.
- Expression evaluation for multi-AU control relies on emotion-level proxies and a limited set of 100 audio-image groups.
- Teeth quality required optional GFP-GAN fine-tuning post-processing.
Experimental setup
Datasets: HDTF CelebV-HQ DISFA MEAD VoxCeleb2 CFD
Benchmarks: HDTF MEAD CelebV-HQ
Metrics: ACC-emo PSNR SSIM LPIPS APD AED MAE user study preference
Baselines: EAMM VideoRetalking TPSM StyleHEAT LIA DPE
Builds on: LIA PIRenderer Wav2Vec 2.0 GFP-GAN MicronBERT
Closest papers by meaning
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.88)
- AnyTalk: Multi-modal Driven Multi-domain Talking Head Generation (AAAI 2025 · similarity 0.88)
- Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Control (ICLR 2025 · similarity 0.88)
- Occlusion-Insensitive Talking Head Video Generation via Facelet Compensation (AAAI 2025 · similarity 0.88)
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.87)
- Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.