Research map / Image generation / Editing and style transfer
FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlowVQTalker, which pairs a normalizing-flow coefficient generator (ExpFlow/PoseFlow with a Student's t mixture latent space) with a VQ-GAN-style codebook image generator to produce diverse, synchronized emotional talking faces with HD textures and clear teeth.
Problem
Existing audio-driven talking face methods are deterministic, producing fixed non-verbal dynamics (expressions, blinks, poses), and low-resolution renderers that fail to capture emotion-aware textures and clear teeth.
Approach
Model emotional 3DMM expression and pose coefficients with conditional normalizing flows whose latent space is a Student's t mixture (one component per emotion), sample latents for diversity, and render frames via a codebook-query VQ generator whose pretrained codebook supplies high-quality textures.
Key results
- On MEAD, FlowVQTalker reports FID 16.553 vs 19.836 for the best prior listed (EAT) and CPBD 0.181 vs 0.120 for EAT.
- On HDTF, FlowVQTalker reports FID 15.165 and CPBD 0.268, versus 18.316 FID and 0.121 CPBD for EAT.
- User study with 20 participants gives FlowVQTalker lip-sync 4.03 and image-quality 4.25, above the other compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FlowVQTalker instead of the strongest compared baseline | MEAD talking-face generation · MEAD | FID: 19.836 -> 16.553 | Helped |
| Use FlowVQTalker instead of the strongest compared baseline | MEAD talking-face generation · MEAD | CPBD: 0.125 -> 0.181 | Helped |
| Use FlowVQTalker instead of the strongest compared baseline | MEAD talking-face generation · MEAD | Accemo: 65.83% -> 71.53% | Helped |
| Use FlowVQTalker instead of the strongest compared baseline | HDTF talking-face generation · HDTF | FID: 16.929 -> 15.165 | Helped |
| Use FlowVQTalker instead of the strongest compared baseline | HDTF talking-face generation · HDTF | CPBD: 0.142 -> 0.268 | Helped |
| Replace SMM with GMM in ExpFlow (ablation) | ExpFlow · MEAD | expression diversity and robustness | Hurt |
| Remove data dropout from ExpFlow (ablation) | ExpFlow · MEAD | expression diversity | Hurt |
| Remove data dropout from ExpFlow (ablation) | ExpFlow · MEAD | audio-lip synchronization | Hurt |
| Add warping network W (ablation) | VQIG baseline · talking-face image generation | spatial transformation | Helped |
| Add fuse network φ (ablation) | VQIG with W · talking-face image generation | identity and texture fidelity | Helped |
| Add multi-head cross-attention (ablation) | VQIG with W and φ · talking-face image generation | identity and texture fidelity | Helped |
| Add codebook C (ablation) | VQIG with W, φ and MHCA · talking-face image generation | image fidelity and clarity | Helped |
Surprising findings
- Wav2Lip achieves higher SyncNet confidence (8.484 on MEAD) than the ground-truth videos (6.733).
Limitations
- Emotional training data is small (MEAD), which the authors say motivates the SMM and data dropout but limits generalization.
- Pose modeling is trained without pose labels in MEAD and uses a Gaussian prior, ignoring emotion influence on pose.
Experimental setup
Datasets: MEAD HDTF FFHQ
Benchmarks: MEAD HDTF
Metrics: SSIM FID M-LMD F-LMD Sync confidence Accemo CPBD user study scores
Baselines: Wav2Lip PC-AVS IP-LAP EAMM EMMN EAT PD-FGC MakeItTalk Audio2Head ACVT SadTalker
Builds on: VQGAN Glow-style normalizing flow MEAD SadTalker
Closest papers by meaning
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.91)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.90)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.90)
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · similarity 0.90)
- VQTalker: Towards Multilingual Talking Avatars Through Facial Motion Tokenization (AAAI 2025 · similarity 0.89)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.