Research map / Video / Video generation
AudCast: Audio-Driven Human Video Generation by Cascaded Diffusion Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AudCast, a two-stage cascade of diffusion transformers that generates holistic audio-driven human videos (lip-sync plus body gestures) from a reference image, with a 3D-prior regional refinement stage for hands and face.
Problem
Audio-driven human video generation has mostly driven facial movements, producing incoherent head-body dynamics and poor hand/face details; generating holistic videos with accurate lip-sync and co-speech gestures from a single reference image is hard.
Approach
A Holistic Human DiT (initialized from CogVideoX-2B, with audio cross-attention, head position guidance, motion tokens, and appearance/identity adapters) generates the full-body video directly from audio, then a Regional Refinement DiT uses 3D face and hand mesh priors to inpaint and sharpen hand and face regions.
Key results
- Ours achieves SSIM 0.8275, LPIPS 0.1368, FVD 527.24 versus Talk-Mimic's 0.5497, 0.4375, 1648.92 on the combined test set.
- On the Vlogger demo set, Ours gets Hand-C 0.8522 versus Vlogger's 0.8407 and Body-C 0.9059 versus 0.9231.
- Removing R2-DiT lowers Hand-C from 0.8832 to 0.8579 and raises LPIPS from 0.1368 to 0.1399 in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AudCast instead of Prob-CNxt | Audio-driven human video generation · Combined test set | FID: 74.20 -> 45.41 | Helped |
| Use AudCast instead of S2D | Audio-driven human video generation for Oliver · Oliver | SSIM: 0.7462 -> 0.7832 | Helped |
| Use AudCast instead of S2D | Audio-driven human video generation for Oliver · Oliver | CosSim: 0.8325 -> 0.7527 | Hurt |
| Use AudCast instead of Vlogger | Audio-driven human video generation · Vlogger demo set | Hand-V×102: 0.1855 -> 0.4037 | Helped |
| Add R2-DiT refinement (ablation) | AudCast · Combined test set | Hand-C: 0.8579 -> 0.8832 | Helped |
| Add Motion Tokens (ablation) | AudCast · Combined test set | FVD: 629.30 -> 527.24 | Helped |
| Add Head Position Encoder (ablation) | AudCast · Combined test set | Body-C: 0.9319 -> 0.9485 | Helped |
| Add Appearance Adapter (ablation) | AudCast · Combined test set | SSIM: 0.7253 -> 0.8275 | Helped |
| Add Identity Adapter (ablation) | AudCast · Combined test set | CosSim: 0.6839 -> 0.7454 | Helped |
Surprising findings
- On the Vlogger demo set, Vlogger has higher Body-C (0.9231) than AudCast (0.9059), and the S2D method obtains better hand dynamics (Hand-V) and identity preservation (CosSim) than AudCast on the Oliver subject.
Limitations
- Baselines are assembled from combinations of gesture generators and pose-driven models, not end-to-end audio-to-video methods, so comparisons are indirect.
- Training data is an in-house dataset that is not released, and centrally positioned speaking humans are the focus.
- Long-video generation relies on chaining independent inferences with motion tokens, which may accumulate errors.
Experimental setup
Datasets: in-house ~300-hour speaking human video dataset PATs video samples (test) Vlogger demo set (test)
Benchmarks: in-house test set PATs test clips Vlogger demo set
Metrics: SSIM LPIPS FID FID-VID FVD Beat Alignment Score Body-C Hand-C Hand-V CosSim
Baselines: TalkSHOW + MimicMotion TalkSHOW + ControlNeXt ProbTalk + MimicMotion ProbTalk + ControlNeXt Vlogger S2G S2D
Builds on: CogVideoX MM-DiT ControlNeXt wav2vec ArcFace
Closest papers by meaning
- Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model (CVPR 2024 · similarity 0.89)
- VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis (CVPR 2025 · similarity 0.88)
- Diffusion-based Realistic Listening Head Generation via Hybrid Motion Modeling (CVPR 2025 · similarity 0.87)
- 🎧MOSPA: Human Motion Generation Driven by Spatial Audio (NeurIPS 2025 · similarity 0.87)
- ShowMaker: Creating High-Fidelity 2D Human Video via Fine-Grained Diffusion Modeling (NeurIPS 2024 · similarity 0.87)
- Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.