Research map / Large language models / Language, safety and interpretability
LLaMA-Omni: Seamless Speech Interaction with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds LLaMA-Omni, an end-to-end speech-in/text+speech-out model on Llama-3.1-8B-Instruct with a frozen Whisper encoder and a non-autoregressive CTC streaming speech decoder, trained on a 200K speech-instruction dataset (InstructS2S-200K).
Problem
Open-source LLMs lack low-latency, high-quality speech interaction; cascaded ASR-LLM-TTS systems add latency and speech-language models trained from scratch need large data and compute.
Approach
Attach a frozen Whisper encoder and trainable speech adaptor to an instruct LLM for direct speech understanding, then add a small NAR CTC-based streaming decoder over LLM hidden states that emits discrete HuBERT units in parallel with the text response, vocoded in chunks, trained in two stages on a speech-instruction dataset built by rewriting text instructions and TTS synthesis.
Key results
- LLaMA-Omni achieves a ChatGPT Score of 3.99 on S2TIF and 3.47 on S2SIF in the offline setting, the highest among compared systems (SpeechGPT 2.98/2.19).
- Streaming latency as low as 236ms (Omega=10), below GPT-4o's reported 320ms average audio latency.
- Training takes about 65 hours on 4 NVIDIA L40 GPUs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLaMA-Omni instead of the strongest listed baseline | Speech instruction-following · InstructS2S-Eval S2TIF | ChatGPT Score: 3.47 -> 3.99 | Helped |
| Use LLaMA-Omni instead of SpeechGPT | Speech instruction-following · InstructS2S-Eval S2SIF offline | ChatGPT Score: 2.19 -> 3.47 | Helped |
| Use LLaMA-Omni instead of SALMONN + Orca | Speech instruction-following · InstructS2S-Eval S2SIF offline | ASR-WER: 3.78 -> 10.82 | Hurt |
| Use LLaMA-Omni instead of SALMONN + Orca | Speech response generation · InstructS2S-Eval S2SIF offline | UTMOS: 3.8286 -> 3.9296 | Helped |
| Set unit chunk size Ω to 10 rather than offline generation (ablation) | LLaMA-Omni · InstructS2S-Eval S2SIF | Total latency: 1753.77 -> 236.18 | Helped |
| Use smaller unit chunks (ablation) | LLaMA-Omni · InstructS2S-Eval S2SIF streaming | UTMOS: 3.8242 -> 3.2304 | Hurt |
| Increase unit chunk size from Ω=10 to Ω=100 (ablation) | LLaMA-Omni · InstructS2S-Eval S2SIF streaming | ChatGPT Score: 3.54 -> 3.49 | Hurt |
| Increase unit chunk size from Ω=10 to Ω=100 (ablation) | LLaMA-Omni · InstructS2S-Eval S2SIF streaming | ASR-WER: 9.84 -> 10.71 | Helped |
Surprising findings
- Cascaded systems with streaming TTS show sharp speech-rate drops and score gaps at low latency, while the end-to-end model keeps prosody stable across latency settings.
Limitations
- Speech decoder trained on only about 1K hours, leading to higher ASR-WER than cascaded industrial TTS systems.
- Evaluation relies on GPT-4o scoring over ASR transcripts, which penalizes speech synthesis errors.
- Speech expressiveness is not addressed.
Experimental setup
Datasets: InstructS2S-200K Alpaca UltraChat Alpaca-Eval
Benchmarks: InstructS2S-Eval
Metrics: ChatGPT Score ASR-WER UTMOS latency speech rate (WPS)
Baselines: SpeechGPT SALMONN + Orca Qwen2-Audio + Orca
Builds on: Llama-3.1-8B-Instruct Whisper CTC HiFi-GAN unit vocoder SpeechGPT
Closest papers by meaning
- Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM (ICML 2025 · similarity 0.89)
- VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction (NeurIPS 2025 · similarity 0.87)
- SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation (NeurIPS 2025 · similarity 0.87)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.87)
- Zebra-Llama: Towards Extremely Efficient Hybrid Models (NeurIPS 2025 · similarity 0.87)
- OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-time Emotional Speech Synthesis (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.