Research map / Video / Video generation
GAIA: Zero-shot Talking Avatar Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GAIA, a data-driven zero-shot talking avatar framework that disentangles motion and appearance with a VAE and generates motion latents from speech with a diffusion model, scaled up to 2B parameters on a filtered 1K-hour dataset.
Problem
Zero-shot talking avatar generation from a single portrait and speech, where prior methods rely on domain priors (warping-based motion, 3DMMs) that limit naturalness and diversity.
Approach
Eliminate domain priors by training a VAE with separate motion (landmark) and appearance encoders to disentangle per-frame motion and appearance, then use a Conformer-based diffusion model to generate the motion latent sequence conditioned on speech and a reference frame's motion latent.
Key results
- Speech-driven GAIA reaches subjective MOS of 4.362 naturalness and 4.332 lip-sync vs. SadTalker's 2.884 and 4.012, with Sync-D 8.528 close to real video's 8.548.
- VAE self-reconstruction FID improves from 18.604 (face-vid2vid) to 15.730 with GAIA 700M VAE.
- Scaling diffusion model from 180M to 1.2B lowers Sync-D from 9.145 to 8.528 (on 1K hours).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase VAE training data from 0.5K to 1K hours at 80M parameters (ablation) | GAIA VAE · Self-Reconstruction | FID: 18.353 -> 17.486 | Helped |
| increase VAE size from 80M to 700M parameters with 1K hours of data (ablation) | GAIA VAE · Self-Reconstruction | FID: 17.486 -> 15.730 | Helped |
| increase VAE size from 700M to 1.7B parameters (ablation) | GAIA VAE · Self-Reconstruction | FID: 15.730 -> 15.886 | Hurt |
| increase diffusion training data from 0.1K to 1K hours at 180M parameters (ablation) | GAIA diffusion model · Speech-driven generation | Sync-D: 9.145 -> 8.913 | Helped |
| increase diffusion model size from 180M to 1.2B parameters with 1K hours of data (ablation) | GAIA diffusion model · Speech-driven generation | Sync-D: 8.913 -> 8.528 | Helped |
| remove motion-appearance disentanglement (ablation) | GAIA · Speech-driven generation | FID: 24.242 -> 140.009 | Hurt |
| remove head-pose conditioning (ablation) | GAIA · Speech-driven generation | Mot.: 4.212 -> 3.942 | Hurt |
| remove diffusion and directly predict motion latent with Conformer (ablation) | GAIA · Speech-driven generation | Mot.: 4.212 -> 2.812 | Hurt |
| predict landmarks instead of motion latent (ablation) | GAIA · Speech-driven generation | FID: 24.242 -> 27.022 | Hurt |
| add speech features to each Conformer block and cross-attend to reference motion latent, rather than use add/add conditioning (ablation) | GAIA diffusion conditioning · Speech-driven generation | Sync-D: 8.989 -> 8.913 | Helped |
| replace Conformer backbone with Transformer (ablation) | GAIA diffusion model · Speech-driven generation | MSI: 1.132 -> 0.623 | Hurt |
| use GAIA rather than SadTalker | speech-driven talking avatar generation · Speech-driven generation | Nat.: 2.884 -> 4.362 | Helped |
Surprising findings
- Removing disentanglement raises FID to 140.009, showing the disentanglement is critical.
- Replacing the Conformer with a Transformer causes large motion jitter (MSI drops from 1.132 to 0.623).
Limitations
- Relies on pre-trained landmark and head pose extractors that may hinder end-to-end learning.
- Diffusion-based generation and diverse head poses leave FID comparable rather than best versus baselines.
- Large internal proprietary dataset not released.
Experimental setup
Datasets: HDTF CC v1 CC v2 internal talking avatar dataset TalkingHead-1KH VoxCeleb2
Benchmarks: TalkingHead-1KH out-domain test set (500 videos) VoxCeleb2
Metrics: MOS naturalness MOS lip-sync MOS jitter MOS visual quality MOS motion diversity FID LPIPS PSNR AKD MSI
Baselines: FOMM HeadGAN face-vid2vid MakeItTalk Audio2Head SadTalker PC-AVS PD-FGC
Builds on: wav2vec 2.0 Conformer SDE-based score diffusion
Closest papers by meaning
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.88)
- InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generation (AAAI 2025 · similarity 0.87)
- Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis (ICLR 2024 · similarity 0.87)
- VQTalker: Towards Multilingual Talking Avatars Through Facial Motion Tokenization (AAAI 2025 · similarity 0.87)
- AnyTalk: Multi-modal Driven Multi-domain Talking Head Generation (AAAI 2025 · similarity 0.86)
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.