Research map / Image generation / Fast diffusion sampling
FADA: Fast Diffusion Avatar Synthesis with Mixed-Supervised Multi-CFG Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills a dual-UNet Stable Diffusion talking-avatar model with PeRFlow, using adaptive mixed-supervised losses over varying-quality data and learnable CFG tokens to replace multi-CFG triple inference, reaching 6 steps with 4.17-12.5x NFE speedup.
Problem
Diffusion-based audio-driven talking avatar synthesis is slow due to many denoising steps and multi-CFG inference over audio and reference-image conditions, and naive diffusion distillation degrades robustness and audio-video correlation.
Approach
Adaptive mixed-supervised distillation weights ground-truth loss on moderate-quality data by the ratio of GT loss to teacher loss, and multi-CFG distillation injects learnable CFG tokens combined linearly with audio/reference CFG scales so the student mimics the three-pass multi-CFG output in one pass.
Key results
- Ours-Balanced (18 NFE) achieves FVD-R 50.36 on CelebV-HQ vs 54.0 for Hallo and 54.71 for EchoMimic, and IQA 3.588 vs 3.505 for Hallo.
- Ours-Fast uses 6 NFE, giving the claimed 4.17-12.5x NFE speedup over baselines (Hallo 80, EchoMimic 60, V-Express 50 NFE-D).
- Multi-CFG learnable-token distillation with 6 steps reaches FVD-R 16.67 on HDTF versus 27.10 without CFG distillation, close to teacher FVD 18.27.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill with mixed-supervised loss and multi-CFG learnable tokens | FADA · HDTF test | FVD-R: 18.27 -> 16.67 | Helped |
| Add multi-CFG distillation to Ours-Balanced (ablation) | FADA · CelebV-HQ test | FVD-R: 50.36 -> 54.69 | Hurt |
| Add multi-CFG distillation to Ours-Balanced (ablation) | FADA · HDTF test | FVD-R: 13.32 -> 16.67 | Hurt |
| Train the teacher on moderate-quality dataset B instead of high-quality dataset A (ablation) | Teacher model · HDTF test | IQA: 3.998 -> 3.885 | Hurt |
| Use PeRFlow instead of TCD for distillation (ablation) | FADA student · HDTF test | E-FID: 1.893 -> 1.477 | Helped |
| Use PeRFlow instead of RFlow for distillation (ablation) | FADA student · HDTF test | FVD-R: 81.10 -> 21.91 | Helped |
| Add ground-truth loss with fixed weight 0.2 to student training on dataset B (ablation) | PeRFlow student · HDTF test | FVD-R: 25.06 -> 13.71 | Helped |
| Use adaptive ground-truth loss weighting with peak and dead thresholds rather than unlimited adaptive weighting (ablation) | PeRFlow student · HDTF test | IQA: 3.846 -> 3.951 | Helped |
| Add a CFG layer to the student model (ablation) | FADA student · HDTF test | FVD-R: 27.10 -> 20.05 | Helped |
| Add learnable tokens to the CFG layer (ablation) | FADA student · HDTF test | FVD-R: 20.05 -> 16.67 | Helped |
| Use CFG time embedding instead of no CFG control (ablation) | FADA student · HDTF test | FVD-R: 27.10 -> 23.24 | Helped |
Surprising findings
- Simply adding moderate-quality data when training the teacher model decreased performance, while the student trained with mixed-supervised distillation improved.
- Directly injecting CFG scale as a timestep-style embedding gave only limited recovery, and the student with learnable tokens surpassed the teacher on FVD.
Limitations
- Relies on a large proprietary internet-sourced dataset (160h high-quality plus 1300h moderate-quality), limiting reproducibility.
- Quality drops slightly with CFG distillation (Ours-Fast has higher FVD/FID/E-FID than Ours-Balanced on some sets).
Experimental setup
Datasets: internal speaker videos from internet (160h high-quality A, 1300h moderate-quality B) HDTF CelebV-HQ Loopy open-set test set
Benchmarks: HDTF CelebV-HQ Loopy open-set test set
Metrics: IQA Sync-D FVD FID E-FID NFE-D
Baselines: Hallo EchoMimic V-Express SadTalker TCD RFlow
Builds on: PeRFlow Loopy Stable Diffusion DDIM TCD
Closest papers by meaning
- VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis (CVPR 2025 · similarity 0.89)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.87)
- IV-mixed Sampler: Leveraging Image Diffusion Models for Enhanced Video Synthesis (ICLR 2025 · similarity 0.87)
- Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency (ICLR 2025 · similarity 0.87)
- Zero-1-to-A: Zero-Shot One Image to Animatable Head Avatars Using Video Diffusion (CVPR 2025 · similarity 0.87)
- MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.