Research map / Image generation / Editing and style transfer
VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VLOGGER, a two-stage diffusion pipeline that generates audio-driven talking/moving human video (face and upper body, including hands) from a single image, trained on the new MENTOR dataset.
Problem
Audio-driven human video synthesis from a single reference image that covers full upper body and gestures, not just cropped faces or lips, with temporal consistency and identity preservation.
Approach
A stochastic audio-to-3D-body-motion diffusion transformer predicts face expression and body pose residuals, which condition a temporally extended Imagen-based diffusion model via dense body renders and warped reference images; variable-length video is produced by temporal outpainting.
Key results
- On HDTF, VLOGGER achieves CPBD 0.621 and NIQE 5.92 versus SadTalker's 0.520 and 6.48, with ArcFace 0.759 vs 0.874 for SadTalker (lower is better).
- On TalkingHead-1KH, VLOGGER achieves the best CPBD (0.575) and ArcFace identity distance (0.881 vs 0.961 for SadTalker), but StyleTalk has better FID (38.98 vs 28.94 for VLOGGER, lower is better for VLOGGER).
- Best-of-8 sampling further improves VLOGGER on HDTF to FID 4.58-level jitter and CPBD 0.634, showing diversity gains from stochastic sampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VLOGGER instead of the strongest baseline | Audio-driven video generation · HDTF | FID: 19.44 -> 18.98 | Helped |
| Use VLOGGER instead of the strongest baseline | Audio-driven video generation · HDTF | CPBD: 0.512 -> 0.621 | Helped |
| Use VLOGGER instead of the strongest baseline | Audio-driven video generation · HDTF | LSE-D: 7.55 -> 8.10 | Hurt |
| Use VLOGGER instead of the strongest baseline | Audio-driven video generation · TalkingHead-1KH | FID: 31.45 -> 28.94 | Helped |
| Use VLOGGER instead of the strongest baseline | Audio-driven video generation · TalkingHead-1KH | CPBD: 0.482 -> 0.575 | Helped |
| Use VLOGGER instead of the strongest baseline | Audio-driven video generation · TalkingHead-1KH | LSE-D: 8.17 -> 8.50 | Hurt |
| Predict a residual Δ over body pose (ablation) | VLOGGER motion generation · MENTOR | FID: 52.27 -> 15.36 | Helped |
| Add temporal loss (ablation) | VLOGGER motion generation · MENTOR | Jitter [mm/s3]: 4.64 -> 3.58 | Helped |
| Add classifier-free guidance (ablation) | VLOGGER motion generation · MENTOR | FID: 16.54 -> 15.36 | Helped |
| Add body controls (ablation) | VLOGGER temporal diffusion model · MENTOR | FID: 16.95 -> 15.36 | Helped |
| Use temporal outpainting (ablation) | VLOGGER temporal diffusion model · MENTOR | Jitter [mm/s3]: 3.70 -> 3.58 | Helped |
| Increase outpainting overlap from 25% to 50% (ablation) | VLOGGER temporal diffusion model · MENTOR | Jitter [mm/s3]: 3.61 -> 3.58 | Helped |
Surprising findings
- Temporal jitter of VLOGGER is second lowest after StyleTalk, which introduces very little face motion.
- On TalkingHead-1KH, StyleTalk achieves a lower FID than VLOGGER, contrary to the image-quality advantage claimed elsewhere.
Limitations
- Relies on a proprietary, internal training dataset (MENTOR) and internal Imagen, limiting reproducibility.
- Iterative temporal outpainting trades quality against running time; overlap choice affects cost.
- Lip sync scores are comparable to baselines, so no clear gain on that axis.
Experimental setup
Datasets: MENTOR HDTF TalkingHead-1KH
Benchmarks: HDTF TalkingHead-1KH MENTOR
Metrics: FID CPBD NIQE LSE-D LME expression std head error ArcFace identity distance jitter
Baselines: MakeItTalk Audio2Head Wang et al. (2020) SadTalker StyleTalk
Builds on: Imagen ControlNet SMPL-X DDIM
Closest papers by meaning
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.89)
- Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation (CVPR 2024 · similarity 0.89)
- Make-Your-Anchor: A Diffusion-based 2D Avatar Generation Framework (CVPR 2024 · similarity 0.89)
- FADA: Fast Diffusion Avatar Synthesis with Mixed-Supervised Multi-CFG Distillation (CVPR 2025 · similarity 0.89)
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · similarity 0.89)
- ShowMaker: Creating High-Fidelity 2D Human Video via Fine-Grained Diffusion Modeling (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.