Research map / Large language models / Alignment and preferences
Aligning Spoken Dialogue Models from User Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 150k+ preference dataset from live full-duplex spoken conversations with Moshi, using LLM-judge (Mistral Large 2) feedback and TTS resynthesis, and applies offline alignment (DPO-LN and variants) to fine-tune Moshi for better QA, safety, and timing.
Problem
Preference alignment methods are built for text LLMs and do not handle real-time full-duplex speech dynamics such as interruptions, interjections, overlaps, and unsegmented turns.
Approach
Derive content-related and timing-related preference pairs from raw organic spoken conversations (transcribed, audio discarded, user turns resynthesized via TTS), then align a full-duplex speech-to-speech model offline, applying DPO-style loss only on text tokens.
Key results
- DPO-LN on Moshi raises average QA accuracy from 36.1 to 39.2 (+3.1) and safety average from 70.9 to 77.8 (+6.9).
- Using only timing-related interruption/silence pairs plus content pairs (All mix) reaches 39.8 QA average on unique-context data, while text-only DPO-LN gives the best QA among stream variants.
- Human evaluation on short ~30s conversations shows the aligned model preferred over base Moshi on coherence, engagement, and relevance; gains shrink in longer conversations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Align Moshi with DPO-LN on the final preference-data mix | Moshi · QA benchmarks | average accuracy: 36{.}1 to 39{.}2 (+3{.}1) | Helped |
| Align Moshi with DPO-LN on the final preference-data mix | Moshi · Safety benchmarks | average accuracy: 70{.}9 to 77{.}8 (+6{.}9) | Helped |
| Transfer DPO-LN alignment using the same preference data | Moshika · QA benchmarks | average accuracy: 37{.}2 to 38{.}2 | Helped |
| Transfer DPO-LN alignment using the same preference data | Moshika · Safety benchmarks | average accuracy: 66{.}2 to 77{.}2 (+11{.}0) | Helped |
| Add audio-token preference probabilities to text-stream DPO-LN (ablation) | Moshi · QA benchmarks | average accuracy: 39{.}2 to 35{.}3 | Hurt |
| Add cross-entropy loss on audio tokens to text-stream DPO-LN (ablation) | Moshi · QA benchmarks | average accuracy: 39{.}2 to 34{.}3 | Hurt |
| Replace DPO-LN with DPO (ablation) | Moshi · QA benchmarks | average accuracy: 39{.}2 to 36{.}2 | Hurt |
| Replace DPO-LN with SimPO (ablation) | Moshi · Safety benchmarks | average accuracy: 77{.}8 to 73{.}1 | Hurt |
| Replace DPO-LN with APO-Zero (ablation) | Moshi · QA benchmarks | average accuracy: 39{.}2 to 39{.}0 | No clear effect |
| Use Type-C timing-only preference pairs (ablation) | Moshi · QA benchmarks | average accuracy: 36{.}1 to 39{.}4 | Helped |
| Use Type-A content-only preference pairs (ablation) | Moshi · QA benchmarks | average accuracy: 36{.}1 to 36{.}7 | No clear effect |
| Set DPO-LN β to 0.5 rather than 0.3 (ablation) | Moshi · QA benchmarks | average accuracy: 39{.}2 to 38{.}4 | Hurt |
Surprising findings
- Optimizing on audio tokens (joint text+audio DPO or CE on audio) reduced QA performance relative to text-only DPO.
- Transferring the preference data to a differently voiced Moshi variant (Moshika) raised replay length considerably and increased WER from 3.78% to 5.88%.
Limitations
- Alignment is offline and single-pass; no online RLHF, and preference data mostly targets the first problematic reply, leaving long conversations less aligned.
- Human evaluation is retrospective and limited to short conversations; gains degrade for longer ones with trade-offs among engagement, relevance and coherence.
- User audio is replaced by TTS resynthesis, losing speaker identity and prosodic cues; only Moshi is evaluated as a model.
Experimental setup
Datasets: Llama Questions TriviaQA WebQuestions ALERT XSTest
Benchmarks: Llama Questions TriviaQA WebQuestions ALERT XSTest
Metrics: accuracy replay length WER human ratings (coherence, engagement, relevance)
Baselines: SpeechGPT Spectron Moshi (base) DPO SimPO APO-Zero
Builds on: Moshi DPO SimPO APO RLAIF
Closest papers by meaning
- SpeechAlign: Aligning Speech Generation to Human Preferences (NeurIPS 2024 · similarity 0.91)
- Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment (NeurIPS 2025 · similarity 0.90)
- Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training (ICLR 2025 · similarity 0.88)
- Aligning Language Models Using Follow-up Likelihood as Reward Signal (AAAI 2025 · similarity 0.88)
- Decoupling Conversational Dynamics in Full-Duplex Spoken Models through Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.