Research map / Large language models / Language, safety and interpretability
NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Next-Token-Pair Prediction (NTPP), a decoder-only dual-channel speech LM that jointly models both speakers' speech tokens per time step, extended to VQ and RVQ tokenizers, trained on Fisher dual-channel dialogue.
Problem
Existing spoken dialogue speech language models either condition on one channel, rely on extra encoders or VAD modules, or lack speaker independence, limiting natural real-time full-duplex dialogue from dual-channel speech.
Approach
Predict the joint next token pair (speaker A and speaker B tokens) at each time step with a decoder-only transformer, using token-pair embeddings, channel embeddings, pair-wise causal masking, and a cyclic depth embedding plus flattened depth tokens to support RVQ tokenizers.
Key results
- NTPP achieves lower turn-taking statistic deviations than dGSLM, LSLM and cascaded baselines (e.g., number of IPU |Δ| 1.3 at T=0.9 vs 1.6 for dGSLM).
- Human evaluation on Fisher: NTPP meaningfulness 4.10 and naturalness 4.42, above SyncLLM (4.10/4.33), Moshi (3.20/3.90) and dGSLM (1.82/3.90).
- Speaker-swap robustness: NTPP shows low variation on test set (e.g., IPU |Δ| 0.18 vs 0.43 for Moshi), and inference latency stays lower than Moshi as turns increase.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NTPP for dual-channel dialogue generation instead of the strongest baseline | Fisher turn-taking generation · Fisher generated dialogues | |ΔIPU| occurrences/min: 1.6 -> 1.3 | Helped |
| Use NTPP for dual-channel dialogue generation instead of the strongest baseline | Fisher turn-taking generation · Fisher generated dialogues | |ΔPause| occurrences/min: 2.9 -> 1.9 | Helped |
| Use NTPP for dual-channel dialogue generation instead of the strongest baseline | Fisher turn-taking generation · Fisher generated dialogues | |ΔGap| occurrences/min: 2. -> 1.5 | Helped |
| Use NTPP for dual-channel dialogue generation instead of the strongest baseline | Fisher turn-taking generation · Fisher generated dialogues | |ΔOverlap| occurrences/min: 1. -> 0.9 | Helped |
| Use NTPP for dual-channel dialogue generation instead of the strongest baseline | Fisher turn-taking generation · Fisher generated dialogues | |ΔIPU| duration/min: 4.1 -> 2.9 | Helped |
| Use NTPP for dual-channel dialogue generation instead of the strongest baseline | Fisher turn-taking generation · Fisher generated dialogues | |ΔPause| duration/min: 3.6 -> 2.5 | Helped |
| Use NTPP for dual-channel dialogue generation instead of the strongest baseline | Fisher turn-taking generation · Fisher generated dialogues | |ΔGap| duration/min: 0.9 -> 0.9 | No clear effect |
| Use NTPP rather than SyncLLM for human-rated naturalness | spoken dialogue generation · Overall human evaluation | Naturalness: 4.10 (0.05) -> 4.15 (0.06) | Helped |
| Pretrain on single-channel audio before dual-channel NTPP (ablation) | NTPP · Dual-channel speech learning | perplexity | Helped |
| Include stage-two NTPP training on dual-channel speech (ablation) | NTPP · Dual-channel speech learning | perplexity | Helped |
| Use an RVQ tokenizer instead of a VQ tokenizer (ablation) | NTPP speech modeling · Speech token training | training loss | Helped |
| Remove ASR text transcripts from pretraining (ablation) | SLM pretraining · Single-channel speech test set | perplexity | Helped |
Surprising findings
- Audio-only continual pretraining (without ASR text) gave lower perplexity than training with text transcripts, suggesting modality interference from text.
Limitations
- Limited availability of large-scale dual-channel speech data (Fisher used, 2,200 hours).
- Evaluation of interruptions relies on synthetic GPT-4-generated scenarios synthesized with ChatTTS.
- Human evaluation involved 25 annotators; GPT-4 judge used for content/style scores.
Experimental setup
Datasets: Fisher CANDOR single-channel speech datasets (~140k hours)
Benchmarks: Fisher CANDOR
Metrics: turn-taking statistics (IPU, pause, gap, overlap occurrences perplexity meaningfulness MOS naturalness MOS speaker-swap robustness inference latency GPT score
Baselines: dGSLM LSLM Cascaded (wav2vec2 + KenLM + Google TTS) Moshi SyncLLM
Builds on: next-token prediction dGSLM LSLM Moshi RoPE
Closest papers by meaning
- Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation (NeurIPS 2024 · similarity 0.88)
- CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations (NeurIPS 2024 · similarity 0.88)
- CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching (NeurIPS 2025 · similarity 0.88)
- Long-Form Speech Generation with Spoken Language Models (ICML 2025 · similarity 0.87)
- Language Model Can Listen While Speaking (AAAI 2025 · similarity 0.87)
- MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.