Research map / 3D vision / 3D generation
DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes multi-round dual-speaker interaction for 3D talking head generation, a unified DualTalk model that generates both speaking and listening facial motion, and a 50-hour dual-channel conversation dataset.
Problem
Existing 3D talking head models model either speaking or listening in isolation, so they cannot capture natural role switching and non-verbal feedback in continuous two-person conversation.
Approach
A unified encoder-interaction-decoder framework takes both speakers' audio and speaker-A's blendshape motion to predict speaker-B's blendshape motion, combining cross-modal temporal enhancement, a dual-speaker transformer interaction module with modal alignment attention, and adaptive expression modulation.
Key results
- On the DualTalk test set, expression FD drops to 11.14 versus 24.61 for the second-best L2L, with over 50% expression accuracy improvement.
- On VOCASET, DualTalk achieves LVE 2.7944e-5 mm and LRP 96.69%, surpassing SelfTalk's LRP by about 5 points.
- User study (30 participants, MOS 1-5) rates DualTalk highest on lip sync (4.164), pose naturalness (4.276), expression richness (4.253), and visual quality (4.088).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DualTalk instead of L2L | 3D talking head generation · DualTalk test | FD EXP (×10³): 24.61 -> 11.14 | Helped |
| Use DualTalk instead of L2L | 3D talking head generation · DualTalk OOD | FD EXP (×10³): 30.49 -> 21.71 | Helped |
| Use DualTalk instead of SelfTalk | audio-driven facial animation · VOCASET | LVE (×10⁻⁵ mm): 3.2238 -> 2.7944 | Helped |
| Use DualTalk instead of DIM | listener response generation · ViCo | FD exp: 23.88 -> 22.27 | Helped |
| Use DualTalk instead of L2L | 3D talking head generation · User study | Lip Sync Accuracy: 3.872 -> 4.164 | Helped |
| add Speaker-A’s speech input (ablation) | DualTalk · DualTalk test | FD EXP (×10³): 23.27 -> 11.14 | Helped |
| add Speaker-A’s expression input (ablation) | DualTalk · DualTalk test | FD EXP (×10³): 28.43 -> 11.14 | Helped |
| use the audio feature extractor instead of MFCC (ablation) | DualTalk · DualTalk test | FD EXP (×10³): 27.42 -> 11.14 | Helped |
| add the Cross-Modal Temporal Enhancer (ablation) | DualTalk · DualTalk test | FD EXP (×10³): 16.31 -> 11.14 | Helped |
| add the Dual-Speaker Interaction Module (ablation) | DualTalk · DualTalk test | FD EXP (×10³): 16.70 -> 11.14 | Helped |
| add Adaptive Expression Modulation (ablation) | DualTalk · DualTalk test | FD EXP (×10³): 13.28 -> 11.14 | Helped |
Limitations
- Evaluation of listener and speaker behavior relies on retrained baselines and a 30-participant user study on 30 clips
- Dataset collection and release status is unclear from the main text; details deferred to supplementary material
Experimental setup
Datasets: DualTalk dataset VOCASET ViCo
Benchmarks: DualTalk dataset test and OOD sets VOCASET ViCo
Metrics: FD P-FD MSE SID rPCC LVE FDD LRP MOS
Baselines: FaceFormer CodeTalker EmoTalk SelfTalk L2L VOCA MeshTalk DiffSpeaker RLHG DIM
Builds on: FaceFormer Wav2Vec 2.0 Learning2Listen
Closest papers by meaning
- INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations (CVPR 2025 · similarity 0.89)
- MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation (NeurIPS 2026 · similarity 0.87)
- GLARE: Generating Listening Heads with Appropriate REactions (NeurIPS 2026 · similarity 0.87)
- NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction (ICML 2025 · similarity 0.87)
- Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics (ICLR 2025 · similarity 0.86)
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.