Research map / Video / Video generation
Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MultiTalk, an audio-driven multi-person conversational video generator built on Wan2.1-I2V-14B that binds multi-stream audio to persons via Label Rotary Position Embedding (L-RoPE), trained with partial-parameter and multi-task training.
Problem
Audio-driven human animation methods handle only single persons and a single audio stream, and suffer from incorrect audio-person binding and weak instruction following when generating multi-person conversational videos.
Approach
Inject multi-stream audio into a DiT video model via audio cross-attention, using L-RoPE: person regions (localized adaptively from reference-to-video self-attention) and each audio stream get matching label-based rotary angles so each audio stream binds to its own person; train only audio cross-attention and adapter with multi-task I2V data to keep instruction following.
Key results
- On HDTF, MultiTalk-single reaches Sync-C 8.54 and FVD 95.99, outperforming the listed talking-head baselines on most metrics.
- On EMTD talking body, MultiTalk-multiple achieves Sync-C 8.34 and FID 31.93 versus EchoMimicV2's 6.31 and 35.99.
- Multi-stream models perform comparably to single-stream models, and L-RoPE label-range choices give similar metrics on MTHM (Sync-C 7.47 to 7.56).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use video labels 0--4 and 20--24, with audio labels 2 and 22, instead of video labels 0--2 and 2--4, with audio labels 1 and 3 (ablation) | MultiTalk L-RoPE · MTHM | Sync-C: 7.47 -> 7.56 | No clear effect |
| use video labels 0--4 and 20--24, with audio labels 2 and 22, instead of video labels 0--2 and 2--4, with audio labels 1 and 3 (ablation) | MultiTalk L-RoPE · MTHM | Sync-D: 7.22 -> 7.13 | No clear effect |
| use video labels 0--4 and 20--24, with audio labels 2 and 22, instead of video labels 0--2 and 2--4, with audio labels 1 and 3 (ablation) | MultiTalk L-RoPE · MTHM | E-FID: 3.22 -> 3.16 | No clear effect |
| train all network parameters instead of only audio cross-attention and audio adapter parameters (ablation) | MultiTalk · Text-prompted video generation | instruction-following ability | Hurt |
| train only audio cross-attention and audio adapter parameters instead of all network parameters (ablation) | MultiTalk · Text-prompted video generation | instruction-following ability | Helped |
| add I2V training with multi-event interaction videos alongside AI2V training (ablation) | MultiTalk · Text-prompted video generation | instruction-following ability | Helped |
| use only talking-head and talking-body data for AI2V training, omitting I2V training (ablation) | MultiTalk · Text-prompted video generation | instruction-following ability | Hurt |
| use multi-stream audio training rather than single-stream audio training (ablation) | MultiTalk · HDTF, CelebV-HQ, and EMTD | overall animation performance | No clear effect |
| concatenate the audio streams directly for cross-attention instead of using adaptive L-RoPE binding (ablation) | MultiTalk · Multi-person conversational video generation | audio-person binding | Hurt |
| add the separate cross-attention results for each audio stream instead of using adaptive L-RoPE binding (ablation) | MultiTalk · Multi-person conversational video generation | audio-person binding | Hurt |
| split video latents into fixed left and right regions for audio injection instead of using adaptive L-RoPE (ablation) | MultiTalk · Multi-person conversational video generation with large person movement | audio-person binding | Hurt |
Surprising findings
- Full-parameter training degraded instruction following and caused hand and object distortion, while training only audio cross-attention preserved it.
Limitations
- Performs better with real audio than synthesized audio in facial expression quality.
- Multi-person evaluation set MTHM is small (40 videos) and not public.
Experimental setup
Datasets: in-house 2K hours single-person talking video in-house 200K multi-event interaction clips in-house 100 hours two-person conversation videos HDTF CelebV-HQ EMTD
Benchmarks: HDTF CelebV-HQ EMTD MTHM
Metrics: Sync-C Sync-D E-FID FID FVD
Baselines: AniPortrait VExpress EchoMimic Hallo3 Sonic Fantasy Talking EchoMimicV2 video concatenation
Builds on: Wan2.1 RoPE Wav2Vec CLIP
Closest papers by meaning
- TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation (NeurIPS 2025 · similarity 0.90)
- AnyTalk: Multi-modal Driven Multi-domain Talking Head Generation (AAAI 2025 · similarity 0.89)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.87)
- INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations (CVPR 2025 · similarity 0.87)
- AudCast: Audio-Driven Human Video Generation by Cascaded Diffusion Transformers (CVPR 2025 · similarity 0.87)
- Co-Speech Gesture Video Generation with Implicit Motion-Audio Entanglement (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.