Research map / Large language models / Language, safety and interpretability
Language Model Can Listen While Speaking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LSLM, a single end-to-end speech language model with a listening channel (streaming vq-wav2vec encoder) and a speaking channel (decoder-only token TTS) that enables full-duplex interruption, comparing early, middle, and late fusion.
Problem
Existing speech language models are turn-based and cannot listen while speaking or handle real-time interruptions; prior full-duplex systems rely on cascaded ASR/TTS over text LLMs.
Approach
Model full duplex as an autoregressive speech-token generator conditioned on a streaming SSL listening channel, with an interruption (IRQ) token that stops speech early; fuse the listening channel into each Transformer block (middle fusion) to balance TTS quality and interruption detection.
Key results
- Middle fusion LSLM achieves WER 4.05% (clean) vs 4.28% for vanilla TTS, with F1 98.00% on command-based interruption.
- Early fusion degrades TTS to WER 33.56%, while late fusion drops noisy-condition F1 to 94.89%.
- Voice-based FDM with unseen speakers reaches F1 85.15% under noise and 95.50% on clean input.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use middle fusion instead of early fusion | LSLM · Command-based FDM, Clean | WER (%): 33.56 -> 4.05 | Helped |
| Use middle fusion instead of late fusion | LSLM · Command-based FDM, Noise | F1 (%): 94.89 -> 97.38 | Helped |
| Use voice-based interruptions from diverse words and unseen speakers rather than command-based interruptions | LSLM Middle Fusion · FDM, Noise | F1 (%): 97.38 -> 85.15 | Hurt |
| Train the speaking backbone from scratch and fix the pre-trained listening encoder rather than continuing training both (ablation) | LSLM middle fusion · Command-based FDM | WER (%): 4.05 -> 4.82 | Hurt |
| Train the speaking backbone from scratch and continue training the pre-trained listening encoder rather than continuing training both (ablation) | LSLM middle fusion · Command-based FDM | WER (%): 4.05 -> 4.67 | Hurt |
| Fix the pre-trained speaking backbone and listening encoder rather than continuing training both (ablation) | LSLM middle fusion · Command-based FDM | WER (%): 4.05 -> 6.64 | Hurt |
| Fix the pre-trained speaking backbone and continue training the pre-trained listening encoder rather than continuing training both (ablation) | LSLM middle fusion · Command-based FDM | WER (%): 4.05 -> 4.64 | Hurt |
| Continue training the pre-trained speaking backbone and fix the pre-trained listening encoder rather than continuing training both (ablation) | LSLM middle fusion · Command-based FDM | WER (%): 4.05 -> 4.46 | Hurt |
Surprising findings
- Early fusion of listening and speaking embeddings severely harms speech generation quality (WER over 33%).
Limitations
- Speech-in speech-out with full duplex is not yet addressed; only a single-layer discrete token TTS.
- Voice-based interruption detection degrades notably under noise and with unseen speakers.
- Model is small (106M parameters) and evaluated only on short command-style interruptions.
Experimental setup
Datasets: LibriTTS LibriTTS-testsetB Speech Commands Dataset SEED-TTS MUSAN
Benchmarks: LibriTTS-testsetB Speech Commands test split
Metrics: WER Precision Recall F1
Baselines: Vanilla TTS LSLM early fusion LSLM late fusion
Builds on: vq-wav2vec VALL-E SpeechGPT LauraGPT
Closest papers by meaning
- A Full-duplex Speech Dialogue Scheme Based On Large Language Model (NeurIPS 2024 · similarity 0.90)
- MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation (NeurIPS 2026 · similarity 0.89)
- SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation (NeurIPS 2025 · similarity 0.88)
- Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering (NeurIPS 2026 · similarity 0.88)
- Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics (ICLR 2025 · similarity 0.88)
- Decoupling Conversational Dynamics in Full-Duplex Spoken Models through Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.