Research map / Large language models / Language, safety and interpretability
Long-Form Speech Generation with Spoken Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpeechSSM, a hybrid state-space (Griffin-based) textless spoken LM that generates multi-minute speech in constant memory, plus the LibriSpeech-Long benchmark and reference-based and LLM-judged long-form metrics.
Problem
Generating coherent speech over multiple minutes with textless spoken language models, which typically degrade past tens of seconds due to token rate, long-sequence training/extrapolation limits, and inference memory costs.
Approach
Initialize a Griffin hybrid state-space LM (RecurrentGemma) on USM-v2 semantic tokens with no explicit position embeddings, train on 4-minute segments, and synthesize acoustic tokens with a speaker-prompted SoundStorm stage; evaluate with reference-based SBERT/Gecko metrics, LLM side-by-side win rates, and time-stratified MOS and coherence.
Key results
- SpeechSSM-9B achieves a 75.0% win rate over SpeechSSM-2B-baseline comparisons is not used; on LibriSpeech-Long 4-minute continuations SpeechSSM-2B wins 50% vs itself and SpeechSSM-9B reaches transcript PPL 3.57 versus 3.75 for 2B and 5.71 for Spirit LM Expressive.
- On 16-minute completions, SpeechSSM-9B achieves 68.3% win rate over SpeechSSM-2B's reference, versus 37.7% for Spirit LM Expressive and 32.8% for TWIST-7B, while SpeechTransformer-2B degrades to 24.9%.
- SpeechSSM-2B reaches >120x the throughput of SpeechTransformer when decoding 16.4k-token sequences in batch; the 2B model decodes 10.9 minutes of speech in just over 100 seconds on a TPU v5e.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the Griffin SSM with a Transformer (ablation) | SpeechSSM-2B · LibriSpeech-Long, 16min completion | PPL: 3.59 -> 5.91 | Hurt |
| Add text LM initialization (ablation) | SpeechSSM-2B · LibriSpeech-Long, 4m completion | PPL: 4.57 -> 3.75 | Helped |
| Add text LM initialization (ablation) | SpeechSSM-2B · LibriSpeech test-clean, 7s continuation | PPL: 6.15 -> 5.76 | Helped |
| Train with 16min rather than 4min segments (ablation) | SpeechSSM-2B · LibriSpeech-Long, 4m completion | PPL: 3.75 -> 3.83 | Hurt |
| Train with 16min rather than 4min segments (ablation) | SpeechSSM-2B · LibriSpeech-Long, 16min completion | PPL: 3.59 -> 3.55 | Helped |
| Train with 30s rather than 4min segments (ablation) | SpeechSSM-2B · LibriSpeech test-clean, 7s continuation | PPL: 5.76 -> 5.73 | No clear effect |
| Train with 16min rather than 4min segments (ablation) | SpeechSSM-2B · LibriSpeech test-clean, 7s continuation | PPL: 5.76 -> 5.84 | Hurt |
| Use a larger 9B model instead of 2B | SpeechSSM · LibriSpeech-Long, 4m completion | PPL: 3.75 -> 3.57 | Helped |
| Use slide-and-prompt extension for TWIST-7B | TWIST-7B · LibriSpeech-Long, 16min completion | PPL: 4.45 vs. 3.39 | Hurt |
| Compare synthesis chunk boundaries with chunk centers | SpeechSSM-2B · Synthesis chunk boundaries | MOS: 4.05±0.07 vs. 4.07±0.07 | No clear effect |
Surprising findings
- No model achieves any win over ground-truth transcripts in side-by-side LLM judging, even though SpeechSSM is preferred over baselines.
- Human ratings at synthesis chunk boundaries and centers were nearly identical (MOS 4.05 vs 4.07), suggesting windowed synthesis boundaries do not measurably hurt naturalness.
Limitations
- Model weights are not released and code release is not stated.
- Even the best SpeechSSM variant does not match ground-truth speech in LLM side-by-side judgments.
- Side-by-side judging relies on a single LLM judge (Gemini 2.0 Flash) and on ASR transcripts.
Experimental setup
Datasets: LibriLight unlab-60k LibriSpeech Common Voice v11.0 Multilingual LibriSpeech VoxPopuli extemporaneous monologue corpus (216k hours)
Benchmarks: LibriSpeech test-clean LibriSpeech-Long test-clean LibriSpeech-Long dev-clean
Metrics: transcript perplexity SBERT similarity Gecko similarity side-by-side win rate speaker similarity N-MOS N-MOS-T SC-L sWUGGY sBLiMP
Baselines: GSLM TWIST-1.3B TWIST-7B Spirit LM Expressive AudioLM VoxtLM SpeechTransformer
Builds on: Griffin TWIST SoundStorm SoundStream RecurrentGemma
Closest papers by meaning
- FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing (NeurIPS 2025 · similarity 0.91)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.89)
- Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.88)
- Large Language Models Are Read/Write Policy-Makers for Simultaneous Generation (AAAI 2025 · similarity 0.88)
- NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction (ICML 2025 · similarity 0.87)
- MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.