Research map / Large language models / Language, safety and interpretability
Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Freeze-Omni connects a streaming speech encoder and a speech decoder to a fully frozen Qwen2-7B-Instruct via a three-stage training recipe using ~60K multi-round Q&A pairs, yielding low-latency duplex speech-to-speech dialogue.
Problem
Speech-to-speech multimodal LLMs usually fine-tune the LLM backbone, which requires million-hour spoken data and causes catastrophic forgetting, leaving a gap between spoken and text QA ability.
Approach
Keep the LLM completely frozen and align speech input and output to it through trainable adapters, prompt embeddings, and a NAR prefix speech decoder that consumes the LLM's hidden states, plus chunk-wise state prediction for duplex interruption.
Key results
- Freeze-Omni (Qwen2-7B frozen) reaches 44.73 on Web Q., 72 on LlaMA Q. and 53.88 on Audio Trivia QA, above Moshi (26.6, 62.3, 22.8) and GLM-4-Voice (32.2, 64.7, 39.1), though below text-only Qwen2-7B-Instruct (45.13, 77.67, 63.93).
- Statistical latency averages 745 ms (median 753 ms, 90th percentile 1020 ms); with an estimated 160-320 ms non-statistical and 200-300 ms network latency, real-scenario average is about 1.2 s.
- With the NAR prefix decoder and pre-network, speech decoder CER drops to 1.69% at top-k=2 versus 3.65% without the pre-network (3.9 without prefix).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Freeze-Omni for spoken question answering instead of Qwen2-7B-Instruct with text input | Qwen2-7B-Instruct · Web Questions | accuracy: 45.13 -> 44.73 | Hurt |
| use Freeze-Omni for spoken question answering instead of Qwen2-7B-Instruct with text input | Qwen2-7B-Instruct · LlaMA Questions | accuracy: 77.67 -> 72 | Hurt |
| use Freeze-Omni for spoken question answering instead of Qwen2-7B-Instruct with text input | Qwen2-7B-Instruct · Audio Trivia QA | accuracy: 63.93 -> 53.88 | Hurt |
| decode with chunk=4 instead of chunk=∞ (ablation) | Freeze-Omni speech input · AISHELL-1 | CER (%): 2.15 -> 2.79 | Hurt |
| remove dynamic chunk training while decoding with chunk=4 (ablation) | Freeze-Omni speech input · AISHELL-1 | CER (%): 2.79 -> 2.48 | Helped |
| add a pre-network between the NAR decoder and LLM embedding layer (ablation) | Speech Decoder w/o Prefix · Speech decoder, top-k=1 | CER (%): 5.27 -> 3.11 | Helped |
| add a pre-network between the NAR decoder and LLM embedding layer (ablation) | Speech Decoder w/o Prefix · Speech decoder, top-k=2 | CER (%): 4.64 -> 2.75 | Helped |
| add a pre-network between the NAR decoder and LLM embedding layer (ablation) | Speech Decoder · Speech decoder, top-k=1 | CER (%): 3.9 -> 2.19 | Helped |
| add a pre-network between the NAR decoder and LLM embedding layer (ablation) | Speech Decoder · Speech decoder, top-k=2 | CER (%): 3.65 -> 1.69 | Helped |
| add a pre-network between the NAR decoder and LLM embedding layer (ablation) | Speech Decoder · Speech decoder, top-k=3 | CER (%): 3.53 -> 1.85 | Helped |
Surprising findings
- The speech decoder with a pre-network and prefix reaches lower CER than the version without prefix, indicating hidden-state conditioning helps speech output quality considerably.
Limitations
- Speech output is evaluated only in a single-speaker setting.
- Large gap remains between spoken QA and text-only Qwen2-7B-Instruct accuracy.
- Training ASR data is internal and not released, limiting reproducibility.
Experimental setup
Datasets: moss-003-sft-data internal Chinese and English speech-text ASR data zero-shot TTS synthesized speech LlaMA-Questions Web Questions Trivia QA aishell-1 test_net test_meeting LibriSpeech dev/test sets
Benchmarks: aishell-1 test_net test_meeting LibriSpeech dev-clean/dev-other/test-clean/test-other LlaMA-Questions Web Questions Trivia QA
Metrics: CER WER accuracy latency (ms)
Baselines: Mini-Omni2 Moshi SpeechGPT Spectron GLM-4-Voice Wav2vec2-base Qwen2-7B-Instruct Helium
Builds on: VALL-E TiCodec dynamic chunk training CTC
Closest papers by meaning
- LLaMA-Omni: Seamless Speech Interaction with Large Language Models (ICLR 2025 · similarity 0.89)
- SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation (NeurIPS 2025 · similarity 0.89)
- VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction (NeurIPS 2025 · similarity 0.88)
- A Full-duplex Speech Dialogue Scheme Based On Large Language Model (NeurIPS 2024 · similarity 0.84)
- MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation (NeurIPS 2026 · similarity 0.84)
- Can Speech LLMs Think while Listening? (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.