Research map / Large language models / Language, safety and interpretability
SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SALMONN-omni, a standalone full-duplex speech LLM that avoids audio codec tokens, using a Mamba streaming encoder, Llama-3-8B LLM with explicit "thinking" state tokens, a CosyVoice2-based streaming synthesizer, and DPO for barge-in handling.
Problem
Full-duplex spoken dialogue requires deciding when to listen and speak, handling barge-ins, backchannels and echo; existing codec-injected full-duplex models need large data and lag on speech modality, while modular or multi-LLM systems suffer from error accumulation and overhead.
Approach
Treats the LLM itself as a full-duplex predictor: it interleaves environment-stream speech embeddings, assistant-stream embeddings and text tokens, and generates special <think>/<shift> tokens to control listening/speaking transitions, with no audio codec tokens in the vocabulary; DPO is then applied for context-dependent barge-ins.
Key results
- Under predicted turn-taking, SALMONN-omni reaches 79.3 / 49.7 / 63.6 S2T accuracy on Llama Q., Web Q., TriviaQA versus 74.2 / 40.8 / 45.1 for Freeze-Omni, averaging 35.9% relative improvement over open-source full-duplex models.
- In oracle turn-taking, SALMONN-omni achieves the best S2T on all four datasets (e.g., 66.0 on TriviaQA, 4.05 on AlpacaEval) among compared turn-based models.
- After DPO, overall barge-in F1 rises from 0.86 (SFT) to 0.90 on the contextual setting, with Moshi reaching F1 0.80 on context-independent barge-ins versus 0.88 for SALMONN-omni.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace implicit thinking with explicit thinking (ablation) | SALMONN-omni · Spoken QA and AlpacaEval | S2T: 76.3 -> 82.0; 50.8 -> 52.4; 62.0 -> 67.5; 67.3 -> 68.3 | Helped |
| Add the assistant stream during Stage 2 (ablation) | SALMONN-omni · AlpacaEval | turn-taking success rate: around 70% -> approximately 90% | Helped |
| Use Implicit-ASR instead of Explicit thinking (ablation) | SALMONN-omni · LibriSpeech test-clean | WER: 2.40 -> 3.48 | Hurt |
| Use Explicit-ASR instead of Explicit thinking (ablation) | SALMONN-omni · LibriSpeech test-clean | WER: 2.40 -> 5.09 | Hurt |
| Use Explicit-NS instead of Explicit thinking (ablation) | SALMONN-omni · LibriSpeech test-clean | WER: 2.40 -> 7.65 | Hurt |
| Increase diversity of thinking labels during listening (ablation) | SALMONN-omni · ASR | WER | Hurt |
| Post-train with DPO for 40 steps (ablation) | SALMONN-omni · Contextual-independent and contextual-dependent barge-ins | overall F1: 0.86 -> 0.90 | Helped |
| Use DPO for 10 steps (ablation) | SALMONN-omni · Contextual-independent and contextual-dependent barge-ins | overall F1: 0.86 -> 0.24 | Hurt |
| Use middle-to-late LLM embeddings rather than early-layer embeddings for speech synthesis (ablation) | SALMONN-omni · Llama Questions | S2S: 70.2 -> 73.6 | Helped |
| Use final-layer embeddings rather than layer 24 for speech synthesis (ablation) | SALMONN-omni · Web Questions | S2T: 49.7 -> 48.6 | Hurt |
| Use predicted rather than oracle turn-taking | SALMONN-omni · Spoken QA and AlpacaEval | S2T: 80.0 -> 79.3; 50.5 -> 49.7; 66.0 -> 63.6; 4.05 -> 4.01 | Mixed |
| Use SALMONN-omni rather than Freeze-Omni for predicted turn-taking | Speech interaction LLMs · Llama Questions, Web Questions, TriviaQA, AlpacaEval | S2S: 56.2 -> 73.6; 27.9 -> 43.7; 28.5 -> 56.0; 2.46 -> 3.22 | Helped |
Surprising findings
- DPO initially makes the model overly conservative (F1 drops to 0.24 after 10 steps at batch size 256) before recovering and surpassing the SFT model.
- Adding more diverse thinking labels in the listening state (e.g., inner thoughts or ASR content) hurts performance, so the simplest explicit <think>/<shift> strategy works best.
Limitations
- Context-dependent barge-in tends toward being interrupted after SFT, requiring DPO and sensitive batch-size/step tuning.
- Emotional speech expression is inconsistent, sometimes producing mismatched emotions.
- Evaluation relies on GPT-based judges and a small in-house set of 100-sample barge-in cases.
Experimental setup
Datasets: LibriSpeech-960h GigaSpeech-M Alpaca-52k Web Questions TriviaQA SQuAD Natural Questions VoiceAssistant-400K UltraChat LibriHeavy Llama Questions AlpacaEval (VoiceBench)
Benchmarks: Llama Questions Web Questions TriviaQA AlpacaEval (VoiceBench)
Metrics: accuracy GPTScore turn-taking success rate F1 score WER
Baselines: Moshi Freeze-Omni GLM-4-Voice Qwen2.5-Omni VITA-1.5 miniCPM-o Kimi-Audio Baichuan-Audio
Builds on: Moshi CosyVoice2 Whisper DPO Mamba
Closest papers by meaning
- Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM (ICML 2025 · similarity 0.89)
- Language Model Can Listen While Speaking (AAAI 2025 · similarity 0.88)
- A Full-duplex Speech Dialogue Scheme Based On Large Language Model (NeurIPS 2024 · similarity 0.88)
- MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation (NeurIPS 2026 · similarity 0.87)
- LLaMA-Omni: Seamless Speech Interaction with Large Language Models (ICLR 2025 · similarity 0.87)
- SALMONN: Towards Generic Hearing Abilities for Large Language Models (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.