Research map / Video / Video generation

Full-Duplex Speech-Motion Model for Dyadic Interaction

NeurIPS 2026 · Video › Video generation · approach: Text-to-motion diffusion

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DyaPlex, a causal full-duplex speech-and-motion model that pairs a frozen PersonaPlex speech tower with a trainable motion tower using dyadic interleaving and time-aligned speech-motion RoPE cross-attention, trained on 4,000 h of Seamless Interaction.

Problem

Existing dyadic speech-motion models are non-causal (diffusion or non-streaming tokenizers) or causal but cannot perceive partner motion, so they cannot support full-duplex, streaming speech-and-motion interaction with a human partner.

Approach

A dual-tower Transformer: a frozen full-duplex speech model supplies per-layer residual-stream hidden states, which a trainable causal motion tower cross-attends to, while both speakers' RVQ motion tokens are interleaved in one autoregressive sequence; time-aligned RoPE on cross-attention ties motion frames to concurrent speech frames.

Key results

• Ours (body-only) achieves FGD 5.6e-3 and P-FD 7.3e-3, vs. Audio2Photoreal 57 and 72 and DualTalk 161 and 163 (in x10^-3 units).
• Removing partner motion raises P-FD from 7.3 to 41 and FGD from 5.6 to 39; Delta-User is +31% for the full model.
• In a user study, participants preferred DyaPlex over DualTalk 97.5% and Audio2Photoreal 66.3%, and over ground truth 29.4%.

Surprising findings

• DualTalk collapses to near-constant body pose when trained on Seamless body data, yielding no detectable motion for BeatAlign.
• Removing cross-attention RoPE gives a higher Delta-User (+31%) than some variants, while the full model ties with it, showing that partner-motion gain alone does not capture quality.

Limitations

• Decodes one body token at a time across 22 codes, which may be suboptimal for speed and deployment on robots.
• Evaluation is teacher-forced with ground-truth speech hidden states and partner motion, unlike the streaming setting for baselines.
• Handles only two-person interactions; polyadic extension left to future work.

Experimental setup

Datasets: Seamless Interaction

Benchmarks: Seamless Interaction test subset (330 pairs)

Metrics: FGD Diversity BeatAlign P-FD Delta-User user study preference

Baselines: Audio2Photoreal DualTalk w/o Self-Attn w/o Partner w/o Cross-Attn w/o Cross-RoPE

Builds on: PersonaPlex Moshi GestureLSM MIBURI Mimi

Closest papers by meaning

• Decoupling Conversational Dynamics in Full-Duplex Spoken Models through Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
• Language Model Can Listen While Speaking (AAAI 2025 · similarity 0.87)
• MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation (NeurIPS 2026 · similarity 0.87)
• INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations (CVPR 2025 · similarity 0.86)
• Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering (NeurIPS 2026 · similarity 0.86)
• $\tau$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains (ICML 2026 · similarity 0.85)