Research map / Image generation / Editing and style transfer
Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Style2Talker generates high-resolution audio-driven talking heads with text-controlled emotion style (latent diffusion over 3DMM coefficients) and picture-controlled art style (coefficient-driven DualStyleGAN variant), with LLM/CLIP-annotated emotion text for MEAD.
Problem
Audio-driven talking head methods focus on lip sync and neglect expressive emotion style and artistic style; existing emotion control needs one-hot labels or emotion videos, and no text-emotion audio-visual dataset exists.
Approach
Two-stage framework: a latent/MLP diffusion model conditioned on CLIP-encoded emotion text, identity image and audio predicts 3DMM expression coefficients; a modified DualStyleGAN with a coefficient-driven flow motion generator, content encoder skip connections and refinement network renders 1024x1024 art-styled video. Emotion text is auto-annotated with GPT-3 and CLIP filtering.
Key results
- On MEAD, Style2Talker reports FID 23.207 and SSIM 0.795, versus 63.222 (best baseline AVCT) and 0.726 (StyleTalk).
- Lowest M-LMD on both datasets (3.317 on MEAD, 3.046 on HDTF) among compared methods.
- Sync_conf is second-best on MEAD (2.847 vs Wav2Lip 3.663) and on HDTF it trails AVCT (3.515 vs 2.734).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Style²Talker instead of the strongest listed baseline | Talking-head generation · MEAD | FID: 63.222 -> 23.207 | Helped |
| Use Style²Talker instead of the strongest listed baseline | Talking-head generation · HDTF | M-LMD: 6.268 -> 3.046 | Helped |
| Use Style²Talker instead of the strongest listed baseline | Talking-head generation · MEAD | Sync_conf: 3.663 -> 2.847 | Hurt |
| Add emotion text and the CLIP text encoder (ablation) | Style²Talker · MEAD | F-LMD: 2.913 -> 2.696 | Helped |
| Add the art path and ModRes block using DualStyleGAN (ablation) | Style²Talker · MEAD | FID: 58.119 -> 23.207 | Helped |
| Use diffusion instead of a conditional GAN (ablation) | Style²Talker Style-E stage · MEAD | Sync_conf: 2.521 -> 2.847 | Helped |
| Add the middle-level skip connection (ablation) | Style²Talker Style-A stage · MEAD | FID: 42.105 -> 23.207 | Helped |
| Add refinement network R (ablation) | Style²Talker Style-A stage · MEAD | SSIM: 0.754 -> 0.795 | Helped |
Limitations
- Comparison setting requires VToonify preprocessing of the identity image, so no existing method handles the combined task directly.
- Sync_conf is not best; trails Wav2Lip on MEAD and AVCT on HDTF.
- Ghosting artifacts from misaligned skip-connection features needed a refinement network.
Experimental setup
Datasets: MEAD HDTF art datasets
Benchmarks: MEAD HDTF
Metrics: SSIM FID M-LMD F-LMD Sync_conf PSNR user study
Baselines: MakeItTalk Wav2Lip Audio2Head PC-AVS AVCT EAMM StyleTalk
Builds on: DualStyleGAN StyleGAN DDIM VToonify DALL-E 2 prior
Closest papers by meaning
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.92)
- FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization (CVPR 2024 · similarity 0.91)
- OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking (NeurIPS 2025 · similarity 0.90)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.90)
- Say Anything with Any Style (AAAI 2024 · similarity 0.90)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.