Research map / Video / Video generation
DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
DAWN generates dynamic-length talking head videos non-autoregressively by pairing a latent-motion audio-to-video flow diffusion model with a VAE-based pose/blink generator and two-stage curriculum learning.
Problem
Diffusion-based talking head generation relies on autoregressive or semi-autoregressive strategies, which are slow, underuse context, accumulate errors, and degrade on long videos.
Approach
Decouples lip motion (learned by a latent-flow diffusion model conditioned on HuBERT audio features) from head pose and blink (generated non-autoregressively by a transformer VAE, PBNet), uses RoPE and local attention for extrapolation, and trains with a two-stage curriculum over fixed-short and random-long clips so that all frames are produced in a single pass.
Key results
- On CREMA, DAWN reports FID 5.77, FVD16 56.33, FVD32 75.82 and CSIM 0.845, the best among compared methods.
- On HDTF, DAWN reports CSIM 0.790 and FVD32 95.64, and it generates 8-second videos in 7.32 s versus 11.42 s for an SAR variant and 19.25 s for a two-temporal-resolution variant.
- Ablations show TCL outperforms either stage alone (DAWN FVD16 52.05 vs 81.84 stage-1-only and 125.75 stage-2-only) and removing PBNet raises FID from 9.60 to 15.20.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DAWN's non-autoregressive generation instead of SAR | Talking-head generation · CREMA | Time(s): 11.42 -> 7.32 | Helped |
| Use DAWN's non-autoregressive generation instead of TTR | Talking-head generation · CREMA | FID: 9.77 -> 5.77 | Helped |
| Use DAWN's non-autoregressive generation instead of SAR | Talking-head generation · CREMA | DR25: 0.307 -> 0.044 | Helped |
| Use two-stage curriculum learning rather than only stage 1 (ablation) | DAWN A2V-FDM · CREMA | FVD16: 81.84 -> 52.05 | Helped |
| Use two-stage curriculum learning rather than only stage 2 (ablation) | DAWN A2V-FDM · CREMA | FID: 13.71 -> 9.68 | Helped |
| Add stage 2 to stage 1 training (ablation) | DAWN A2V-FDM · CREMA | FID: 7.95 -> 9.68 | Hurt |
| Add stage 2 to stage 1 training (ablation) | DAWN A2V-FDM · CREMA | FVD16: 81.84 -> 52.05 | Helped |
| Add PBNet to generate pose and blink separately (ablation) | DAWN · HDTF | FID: 15.20 -> 9.60 | Helped |
| Set local attention window to 80 rather than 20 (ablation) | DAWN A2V-FDM · CREMA | FVD16: 159.19 -> 52.05 | Helped |
| Set local attention window to 80 rather than 40 (ablation) | DAWN A2V-FDM · CREMA | FVD16: 72.93 -> 52.05 | Helped |
| Increase local attention window from 80 to 200 (ablation) | DAWN A2V-FDM · CREMA | FVD16: 52.05 -> 53.48 | Hurt |
| Remove local attention (ablation) | DAWN A2V-FDM · CREMA | FVD16: 52.05 -> 63.95 | Hurt |
Surprising findings
- Hallo's degradation rate rises sharply with longer inference lengths, while DAWN's stays stable, suggesting AR-based methods accumulate errors.
- Removing the local attention mechanism gives only modest degradation, which the authors attribute to TCL and the model structure supporting extrapolation.
Limitations
- Cannot model physical common sense, so items like hats or headpieces may not move with the head and cause artifacts.
- Sub-modules must be trained separately; no joint training, so errors propagate across modules.
- Evaluated only on 128x128 resolution for main results, with 256x256 showing higher FID (11.80 vs 9.60).
Experimental setup
Datasets: CREMA HDTF VoxCeleb
Benchmarks: CREMA HDTF
Metrics: FID FVD16 FVD32 LSE-C LSE-D CSIM BAS Blink/s Degradation Rate user study scores
Baselines: Wav2Lip Audio2Head SadTalker Diffused Heads DreamTalk Hallo EchoMimic
Builds on: Latent Flow Generator (motion-based latent warping) MRAA 3D U-Net video diffusion RoPE VAE
Closest papers by meaning
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.91)
- Diffusion-based Realistic Listening Head Generation via Hybrid Motion Modeling (CVPR 2025 · similarity 0.90)
- Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Model (ICML 2025 · similarity 0.90)
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · similarity 0.90)
- AnyTalk: Multi-modal Driven Multi-domain Talking Head Generation (AAAI 2025 · similarity 0.90)
- VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.