Research map / Large language models / Language, safety and interpretability
A Full-duplex Speech Dialogue Scheme Based On Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a full-duplex LLM-based speech dialogue system that uses a two-state neural FSM (SPEAK/LISTEN) with control tokens, fine-tuning Llama-3-8B-Instruct on GPT-4-generated interruption dialogues to cut response latency over half-duplex baselines.
Problem
Existing chat LLM speech dialogue systems are half-duplex (round-based), causing high response latency and poor interruption handling; full-duplex LLM dialogue has not been systematically realized.
Approach
An LLM is aligned to operate a two-state neural finite state machine via control tokens emitted in a serialized token tape, with ASR perception and TTS motor modules running in parallel, so turn-taking and interruption become next-token prediction.
Key results
- Full-duplex system reduces average first token emission delay from 2.28s (half-duplex) to 0.68s, a more than 3x reduction.
- Llama-3-8B-Instruct-fd achieves interruption precision 54.7% vs 46.6% for GPT-4o and 24.7% for GPT-3.5-turbo-0125, and average proper response rate 96.7%.
- Fine-tuning changes OpenCompass scores only slightly (e.g., MMLU 68.4 to 67.5, GSM-8K 79.3 to 75.3, HumanEval 55.5 to 50.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the full-duplex configuration with streaming ASR and streaming TTS instead of the half-duplex non-streaming setup | Llama-3-8B-Instruct dialogue system · Full-duplex dialogue simulator | FTED average latency: 2.28s -> 0.68s | Helped |
| Use semi-streaming ASR with a duplex LLM and non-streaming TTS instead of the non-streaming baseline | Llama-3-8B-Instruct dialogue system · Full-duplex dialogue simulator | FTED average latency: 2.28s -> 1.49s | Helped |
| Use streaming ASR with a duplex LLM and non-streaming TTS instead of the non-streaming baseline | Llama-3-8B-Instruct dialogue system · Full-duplex dialogue simulator | FTED average latency: 2.28s -> 1.15s | Helped |
| Use streaming TTS with the streaming-ASR duplex configuration | Configuration 3: asr-s+llm-fd+tts-ns · Full-duplex dialogue simulator | FTED average latency: 1.15s -> 0.68s | Helped |
| Fine-tune the base model on full-duplex dialogue data | Llama-3-8B-Instruct · Machine interrupts user | Precision: 37.4 -> 54.7 | Helped |
| Fine-tune the base model on full-duplex dialogue data | Llama-3-8B-Instruct · Machine interrupts user | PIR_mid: 36.1 -> 79.1 | Helped |
| Fine-tune the base model on full-duplex dialogue data | Llama-3-8B-Instruct · Machine interrupts user | PRR_mid: 79.3 -> 91.3 | Helped |
| Fine-tune the base model on full-duplex dialogue data | Llama-3-8B-Instruct · User interrupts machine | PRR_avg: 93.6 -> 96.7 | Helped |
| Fine-tune the base model on full-duplex dialogue data | Llama-3-8B-Instruct · MMLU | score: 68.4 -> 67.5 | Hurt |
| Fine-tune the base model on full-duplex dialogue data | Llama-3-8B-Instruct · TriviaQA | score: 64.8 -> 65.9 | Helped |
| Fine-tune the base model on full-duplex dialogue data | Llama-3-8B-Instruct · HumanEval | score: 55.5 -> 50.6 | Hurt |
| Fine-tune the base model on full-duplex dialogue data | Llama-3-8B-Instruct · GSM-8K | score: 79.3 -> 75.3 | Hurt |
Surprising findings
- GPT-4o and GPT-3.5-turbo rarely interrupt mid-sentence, waiting for clear stopping cues, which increases latency.
- Llama-3-8B-Instruct interrupts incomplete input in 75% of cases but with reasonableness below 40%.
Limitations
- Relies on cascaded ASR and TTS modules, adding transmission latency, unlike end-to-end audio LLMs.
- Effects of full-duplex training on response usefulness, relevance, and safety not evaluated in detail.
- Evaluation is largely simulated with GPT-4-turbo judging, not real human interaction.
Experimental setup
Datasets: ShareGPT GPT-4-generated dialogues
Benchmarks: duplex-dialogue-3k OpenCompass MMLU TriviaQA HumanEval GSM-8K MATH
Metrics: first token emission delay proper interruption rate proper response rate interruption precision recall latency average/50%/90%
Baselines: GPT-4o GPT-3.5-turbo-0125 Llama-3-8B-Instruct half-duplex ASR+LLM+TTS pipeline
Builds on: instruction tuning supervised fine-tuning Whisper VITS
Closest papers by meaning
- Language Model Can Listen While Speaking (AAAI 2025 · similarity 0.90)
- SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation (NeurIPS 2025 · similarity 0.88)
- MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation (NeurIPS 2026 · similarity 0.87)
- NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction (ICML 2025 · similarity 0.86)
- Decoupling Conversational Dynamics in Full-Duplex Spoken Models through Reinforcement Learning (NeurIPS 2026 · similarity 0.86)
- From Retrieval to Generation: A Simple and Unified Generative Model for End-to-End Task-Oriented Dialogue (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.