Research map / Large language models / Language, safety and interpretability
Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes USDM, an LLM-based end-to-end spoken dialog model that generates spoken responses with prosody by using prosody-carrying acoustic units, unified speech-text pretraining, and a multi-step (ASR→text response→speech units) fine-tuning template.
Problem
Spoken dialog modeling with LLMs: generating coherent spoken responses with natural, context-appropriate prosody without explicit ASR/TTS cascades, which suffer from linguistic discrepancy and error propagation.
Approach
Shows that k=10,000 XLS-R acoustic units carry emotion and pitch information; pretrains Mistral-7B on interleaved speech-text sequences built with random per-segment modality selection plus correspondence/continuation special tokens; fine-tunes with a transcribe-then-respond-in-text-then-speak template and decodes units with a Voicebox-based unit-to-speech model.
Key results
- USDM achieves MOS 4.31 and P-MOS 4.31, above Cascaded (4.26 / 4.22) and SpeechGPT (4.08 / 4.04) on DailyTalk.
- USDM reaches METEOR 13.1 and ROUGE-L 15.7 on transcribed responses, versus 12.5/15.0 for Cascaded and 9.9/11.8 for SpeechGPT.
- Unit emotion classification on CREMA-D reaches 60.8% accuracy against 16.6% chance, indicating paralinguistic cues in acoustic units.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add unified pretraining that models continuation and correspondence relationships (ablation) | USDM · DailyTalk | METEOR: 8.9 -> 13.1 | Helped |
| add unified pretraining that models continuation and correspondence relationships (ablation) | USDM · DailyTalk | STT WER: 57.8\% -> 7.4\% | Helped |
| add correspondence relationship modeling to continuation-only pretraining (ablation) | USDM pretraining · DailyTalk | STT WER: 57.8\% -> 7.4\% | Helped |
| add continuation relationship modeling to correspondence-only pretraining (ablation) | USDM pretraining · LibriSpeech test-clean | Text PPL: 31.679 -> 6.886 | Helped |
| replace unified pretraining with the fixed Setup 3 template (ablation) | USDM pretraining · DailyTalk | METEOR: 13.1 -> 12.7 | Hurt |
| remove intermediate text modeling and generate speech directly from speech (ablation) | USDM · DailyTalk | METEOR: 13.1 -> 6.5 | Hurt |
| use the ground-truth input transcript instead of the model-generated transcript (ablation) | USDM · DailyTalk | METEOR: 13.1 -> 13.6 | Helped |
| use the ground-truth input transcript instead of the model-generated transcript (ablation) | USDM · DailyTalk | ROUGE-L: 15.7 -> 16.2 | Helped |
| add speech-text pretraining rather than fine-tuning Mistral-7B directly on spoken dialogs (ablation) | USDM · DailyTalk | TTS WER: 64.0\% -> 2.0\% | Helped |
| increase LoRA fine-tuning rank from 8 to 256 (ablation) | USDM fine-tuning · DailyTalk | semantic performance | Helped |
| increase LoRA fine-tuning rank from 8 to 256 (ablation) | USDM fine-tuning · DailyTalk | unit-to-text task performance | Hurt |
| use acoustic units as input to emotion recognition | 3-layer emotion classifier · CREMA-D | accuracy: 16.6\% -> 60.8\% | Helped |
Surprising findings
- The Cascaded baseline has a much lower ASR WER (3.8%) than USDM (7.4%), yet USDM yields more semantically coherent responses.
- Without speech-text pretraining (From Scratch), the model mostly fails to produce speech matching the intermediate text, with TTS WER of 64.0%.
Limitations
- Pretraining explored only with Mistral-7B and English data; generalization to other LLMs and languages untested.
- Requires intermediate text generation; cannot yet directly map input speech to output speech without cross-modal chaining.
- Overfitting in unit-to-text task when fine-tuning with more parameters (LoRA rank analysis).
Experimental setup
Datasets: Multilingual LibriSpeech People's Speech GigaSpeech Common Voice 15.0 VoxPopuli DailyTalk CREMA-D LibriSpeech test-clean Expresso Fisher
Benchmarks: DailyTalk
Metrics: MOS P-MOS human preference win/tie/lose METEOR ROUGE-L STT WER TTS WER GPT-4 semantic preference perplexity on LibriSpeech test-clean
Baselines: Cascaded (whisper-large-v3 + Mistral-7B + Voicebox) From Scratch SpeechGPT-7B-cm
Builds on: Mistral-7B XLS-R Voicebox BigVGAN SpeechGPT
Closest papers by meaning
- SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.89)
- NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction (ICML 2025 · similarity 0.88)
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark (ICLR 2026 · similarity 0.88)
- DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American English (NeurIPS 2026 · similarity 0.88)
- A Variational Framework for Improving Naturalness in Generative Spoken Language Models (ICML 2025 · similarity 0.88)
- Scaling Speech-Text Pre-training with Synthetic Interleaved Data (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.