Research map / Video / Video generation
VQTalker: Towards Multilingual Talking Avatars Through Facial Motion Tokenization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VQTalker, a talking-head framework that quantizes facial motion with Group Residual FSQ and generates motion tokens from CosyVoice speech tokens via a BERT coarse-to-fine model, targeting multilingual lip sync at 512x512 and about 11 kbps.
Problem
Talking head generation performs poorly on non-Indo-European languages because training data is dominated by Indo-European languages and continuous representations overfit; existing methods also have high bitrate and limited resolution.
Approach
Discretize facial motion with a Group Residual Finite Scalar Quantization tokenizer that decouples motion from texture, then generate residual motion tokens coarse-to-fine with a BERT model conditioned on discrete speech tokens, using non-autoregressive generation within each layer and autoregressive generation across residual layers.
Key results
- On HDTF video reconstruction at 512x512, VQTalker reaches about 11 kbps with SSIM 0.874 and LPIPS 0.083, versus 16 kbps for LIA/AniTalker and about 347 kbps for H.264.
- On the MNTE non-Indo-European set, VQTalker gets FID 43.411 and FVD 233.586 versus 55.070 and 321.825 for the next best (EDTalker).
- Discrete audio and motion tokens (CosyVoice + GRFSQ) beat continuous Whisper + vector representations in the ablation (SSIM 0.835 vs 0.795).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VQTalker instead of the strongest baseline | speech-driven talking head generation · HDTF (Indo-European) | SSIM: 0.823 -> 0.835 | Helped |
| use VQTalker instead of the strongest baseline | speech-driven talking head generation · HDTF (Indo-European) | CSIM: 0.881 -> 0.902 | Helped |
| use VQTalker instead of the strongest baseline | speech-driven talking head generation · HDTF (Indo-European) | LMD: 0.084 -> 0.067 | Helped |
| use VQTalker instead of the strongest baseline | speech-driven talking head generation · MNTE (Non-Indo-European) | SSIM: 0.745 -> 0.730 | Hurt |
| use VQTalker instead of the strongest baseline | speech-driven talking head generation · MNTE (Non-Indo-European) | CSIM: 0.802 -> 0.809 | Helped |
| use VQTalker instead of the strongest baseline | speech-driven talking head generation · MNTE (Non-Indo-European) | LMD: 0.107 -> 0.146 | Hurt |
| replace VQTalker with AniTalker | video reconstruction · HDTF | SSIM: 0.905 -> 0.874 | Hurt |
| replace VQTalker with AniTalker | video reconstruction · HDTF | LPIPS: 0.083 -> 0.079 | Helped |
| add FSQ to GRVQ (ablation) | GRFSQ quantizer · HDTF cross-identity driven | MSI: 0.359 -> 0.992 | Helped |
| add FSQ to GRVQ (ablation) | GRFSQ quantizer · HDTF cross-identity driven | Codebook utilization: 26.78 -> 78.74 | Helped |
| use 8 groups instead of 12 (ablation) | GRFSQ quantizer · HDTF cross-identity driven | MSI: 0.992 -> 1.016 | Helped |
| use 2 residual layers instead of 4 (ablation) | GRFSQ quantizer · HDTF cross-identity driven | MSI: 0.992 -> 0.616 | Hurt |
Surprising findings
- VQTalker does not reach the best score on several reconstruction metrics (SSIM and LPIPS are second-best), which the authors attribute to quantization information loss.
- The user study shows EchoMimic has higher visual appeal than VQTalker.
Limitations
- Slight jitter under extreme facial movements.
- Background and accessory edges blur because of the warping method.
- Training data is about 97% Indo-European, and the non-Indo-European evaluation set is small (30 videos).
Experimental setup
Datasets: VoxCeleb HDTF VFHQ
Benchmarks: HDTF MNTE
Metrics: SSIM LPIPS CSIM CPBD LMD FID FVD MSI bitrate
Baselines: SadTalker EAT PD-FGC AniTalker EDTalker EchoMimic LivePortrait EMOPortraits FOMM DPE MTIA Vid2Vid
Builds on: LIA FSQ Residual VQ Group VQ CosyVoice
Closest papers by meaning
- FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization (CVPR 2024 · similarity 0.89)
- AnyTalk: Multi-modal Driven Multi-domain Talking Head Generation (AAAI 2025 · similarity 0.88)
- Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation (CVPR 2025 · similarity 0.87)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.87)
- ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.87)
- Faces that Speak: Jointly Synthesising Talking Face and Speech from Text (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.