Research map / Image generation / Autoregressive and latent generation
INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes INFP, a two-stage audio-driven framework that generates interactive talking/listening head videos from dyadic (dual-track) audio without explicit role switching, and releases the DyConv dataset.
Problem
Audio-driven head generation has focused on single-sided talking or listening, or on dyadic generation that requires manual role assignment and explicit role switching; natural bidirectional conversational agents need seamless speaking/listening transitions.
Approach
A motion-latent space learned from real conversation videos (with facial pixel masking and hybrid contour representation) animates a portrait; a conditional diffusion transformer maps dual-track audio (agent and partner) to motion latents via an interactive motion guider with learnable verbal/non-verbal memory banks, letting the agent switch between speaking and listening implicitly.
Key results
- On DyConv, INFP reaches SyncScore 7.188 and FID 15.727 vs DIM's 4.778 and 34.361, with SID 2.613 vs 0.766.
- In a 20-participant MOS study, INFP scores 4.38 naturalness and 4.49 motion diversity vs DIM's 2.71 and 2.14.
- On HDTF talking-head generation, INFP achieves the best FID (18.582) and CSIM (0.886) and top SyncScore (6.670) among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use INFP instead of DIM | interactive head generation · DyConv test set | SSIM: 0.651 -> 0.834 | Helped |
| use INFP instead of DIM | interactive head generation · DyConv test set | SyncScore: 4.778 -> 7.188 | Helped |
| use INFP instead of DIM | interactive head generation · DyConv user study | Naturalness: 2.71 -> 4.38 | Helped |
| use INFP instead of DIM | listening head generation · ViCo test set | MSE (expression): 0.70 -> 0.51 | Helped |
| use INFP instead of EchoMimic | talking head generation · HDTF test set | FID: 20.873 -> 18.582 | Helped |
| use INFP instead of EchoMimic | talking head generation · HDTF test set | SSIM: 0.807 -> 0.804 | Hurt |
| add verbal and non-verbal motion memory banks (ablation) | INFP interactive motion guider · DyConv test set | SyncScore: 6.103 -> 7.188 | Helped |
| add motion style vector and modulation layers (ablation) | INFP · DyConv user study | Naturalness: 4.02 -> 4.38 | Helped |
| use hybrid facial representation instead of intact image (ablation) | INFP motion encoder · DyConv test set | SSIM: 0.802 -> 0.834 | Helped |
| use hybrid facial representation instead of 2D landmarks map (ablation) | INFP motion encoder · DyConv test set | SSIM: 0.821 -> 0.834 | Helped |
Surprising findings
- Explicit role-switching designs are argued to cause unnatural transitions, and the paper reports baseline DIM fails to produce reasonable lip movement or response feedback in dyadic settings.
Limitations
- Only conversation audio is used as input; no visual or textual control signals.
- Generation is limited to the head region; extension to upper or whole body is left to future work.
- Potential misuse for deceptive talking videos; core model access is to be restricted to research institutions.
Experimental setup
Datasets: DyConv ViCo HDTF
Benchmarks: DyConv ViCo HDTF
Metrics: SSIM PSNR FID SyncScore LPIPS CSIM SID Var FD MSE
Baselines: DIM L2L RLHG SadTalker AniTalker EchoMimic
Builds on: DIM AnimateDiff StyleGAN2 VQ-VAE
Closest papers by meaning
- DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversations (CVPR 2025 · similarity 0.89)
- GLARE: Generating Listening Heads with Appropriate REactions (NeurIPS 2026 · similarity 0.88)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.87)
- Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (NeurIPS 2025 · similarity 0.87)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.87)
- OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.