Research map / Reinforcement learning / RL from feedback

Decoupling Conversational Dynamics in Full-Duplex Spoken Models through Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DuplexPO, a GRPO-style RL method that optimizes only turn-taking/backchannel/barge-in timing decisions over dynamics-critical windows of a full-duplex spoken LLM, using a factorized event-level reward (FCDR), while preserving instruction-following and reasoning.

Problem

Full-duplex spoken dialogue models gain conversational dynamics (low latency, backchannels, barge-in handling) but lose reasoning and instruction-following ability; coupling "what to say" and "when to speak" in one objective is hypothesized to cause this intelligence-dynamics trade-off.

Approach

Decouple timing from content: sample short windows around annotated agent speaking events from human conversations, score the sampled frame-level speaking decisions with a factorized event-level reward (onset timing, backchannel timing, yielding after barge-in, regularization), and update the policy with a group-normalized (GRPO-style) objective with KL to the SFT reference.

Key results

• On Fisher, DuplexPO reaches 0.69 onset MAE and 98.7% turn-taking yield rate versus 0.98 and 92.1% for the SFT Baseline.
• On FDB-v3, DuplexPO achieves 0.24 s turn-taking latency and 100% yield rate, with 5.0% VIR.
• DuplexPO preserves or slightly improves intelligence benchmarks, e.g., OpenBookQA 73.7 vs 72.2 and MMSU 56.2 vs 54.9 over the SFT Baseline.

Surprising findings

• SFT on dynamics-aware dialogue data (SFT Dynamics) regresses on most dynamics metrics relative to the SFT Baseline, indicating SFT alone is insufficient.

Limitations

• Reward ignores pragmatic factors such as user intent, discourse content, speaker style, and culture-specific timing.
• Human timing annotations are not unique ground truth and vary across speakers and languages.
• Policy updates restricted to local windows may miss long-range dialogue effects.

Experimental setup

Datasets: Fisher Seamless-Naturalistic-HQ LibriTTS YODAS Hifi-TTS MUSAN Freesound NeMo ASRSET

Benchmarks: Fisher Seamless Full-Duplex-Bench v3 Llama Questions WebQuestions TriviaQA SDQA AlpacaEval CommonEval OpenBookQA MMSU

Metrics: onset MAE initiation rate yield rate turn-take latency VIR QA accuracy GPT-score

Baselines: Moshi PersonaPlex SFT Baseline SFT Dynamics GPT-Realtime Gemini Live 2.5 Freeze-Omni SALMONN-omni

Builds on: GRPO ORISE Nemotron-VoiceChat CosyVoice2

Closest papers by meaning

• Memory-T1: Reinforcement Learning for Temporal Reasoning in Multi-session Agents (ICLR 2026 · similarity 0.89)
• Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training (ICLR 2025 · similarity 0.88)
• $\tau$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains (ICML 2026 · similarity 0.88)
• Aligning Spoken Dialogue Models from User Interactions (ICML 2025 · similarity 0.88)
• Language Model Can Listen While Speaking (AAAI 2025 · similarity 0.87)
• DialogXpert: Driving Intelligent and Emotion-Aware Conversations Through Online Value-Based Reinforcement Learning with LLM Priors (AAAI 2026 · similarity 0.87)