Research map / 3D vision / Face and shape
VASA-3D: Lifelike Audio-Driven Gaussian Head Avatars from a Single Image
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates an audio-driven, animatable 3D Gaussian head avatar from a single portrait by training on VASA-1-synthesized videos conditioned on VASA-1 motion latents, with SDS and render-consistency losses.
Problem
Creating lifelike, audio-driven 3D head avatars from a single portrait image while capturing subtle expression details, which parametric 3DMM/FLAME models and prior one-shot methods lack.
Approach
Uses the VASA-1 motion latent (learned from 2D videos of ~9.5K subjects) as the driving signal for a FLAME-bound 3D Gaussian head, adds dense latent-conditioned residual deformations, and trains on VASA-1-synthesized multi-pose frames with robust losses (LPIPS, adversarial, SDS over random views, cross-view render consistency).
Key results
- VASA-3D achieves S_C 8.121 and S_D 6.9300 versus best baseline TalkingGaussian 6.701 and 8.1061 for audio-driven lip sync, and was preferred in 93.91% of user study cases.
- In video-driven reenactment on 26 CelebVHQ portraits, VASA-3D reaches PSNR 26.21 and LPIPS 0.0760, outperforming GAGAvatar (25.74, 0.0829).
- Renders 512x512 free-viewpoint video at 75 fps on a single RTX 4090 with 65 ms latency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add VAS deformation driven by VASA motion latents (ablation) | VASA-3D · test set | image quality and lip-sync metrics | Helped |
| add SDS loss (ablation) | VASA-3D · side views | visual artifacts | Helped |
| add SDS loss (ablation) | VASA-3D · rendered views | rendering detail | Hurt |
| add render consistency loss (ablation) | VASA-3D with SDS loss · rendered views | rendering detail | Helped |
| add CAS sharpening loss (ablation) | VASA-3D · rendered images | LPIPS | Helped |
| add CAS sharpening loss (ablation) | VASA-3D · rendered images | PSNR, L1, and SSIM | Hurt |
| increase synthetic training dataset size to 2 hours (ablation) | VASA-3D · test set | PSNR | Helped |
| increase training iterations beyond 200K (ablation) | VASA-3D · test set | PSNR | No clear effect |
| train for 20K instead of 200K iterations (ablation) | VASA-3D · audio-driven generation | S_C: 8.121 -> 7.980 | Hurt |
| train for 20K instead of 200K iterations (ablation) | VASA-3D · audio-driven generation | S_D: 6.9300 -> 7.0020 | Hurt |
| use VASA-3D instead of VASA-1 | audio-driven generation · audio-driven generation | FID: 5.24 -> 7.45 | Hurt |
| use VASA-3D instead of TalkingGaussian | audio-driven 3D talking head generation · audio-driven generation | S_C: 6.701 -> 8.121 | Helped |
Surprising findings
- The 20K-iteration variant of VASA-3D is only slightly worse than the 200K default on lip-sync, suggesting training can be shortened substantially.
Limitations
- Does not model the back of the head due to limited viewing angles of synthetic training videos.
- Does not handle dynamic accessories; upper body not covered.
- Synthetic frames from VASA-1 lack temporal texture consistency; generated faces are detectable as fake by forgery detectors.
Experimental setup
Datasets: VoxCeleb2 CelebVHQ StyleGAN2-generated portraits web audio clips
Benchmarks: VoxCeleb2 test-set VASA-1 synthetic videos CelebVHQ 1-minute real talking videos
Metrics: PSNR PSNR_Face L1 SSIM LPIPS FID SyncNet confidence S_C SyncNet distance S_D ArcFace ID similarity user study satisfaction and preference
Baselines: ER-NeRF GeneFace MimicTalk TalkingGaussian GAGAvatar GPAvatar Real3DPortrait Voodoo3D Portrait4D-v2 VASA-1
Builds on: VASA-1 GaussianAvatars 3D Gaussian Splatting Score distillation sampling (DreamFusion) FLAME
Closest papers by meaning
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · similarity 0.92)
- Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis (ICLR 2024 · similarity 0.90)
- Generalizable and Animatable Gaussian Head Avatar (NeurIPS 2024 · similarity 0.88)
- IM-Portrait: Learning 3D-aware Video Diffusion for Photorealistic Talking Heads from Monocular VideosC (CVPR 2025 · similarity 0.88)
- Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic Gaussians (CVPR 2024 · similarity 0.88)
- High-Fidelity 3D Head Avatars Reconstruction through Spatially-Varying Expression Conditioned Neural Radiance Field (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.