Research map / Large language models / Language, safety and interpretability
Scaling Speech-Text Pre-training with Synthetic Interleaved Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scales speech-text pre-training to 1T tokens by synthesizing 600B tokens of speech-text interleaved data from text corpora via a text-to-token model, with a supervised 12.5Hz VQ speech tokenizer, then fine-tunes into a speech-to-speech chatbot.
Problem
SpeechLMs are limited by the scarcity of unsupervised speech and parallel speech-text data compared with text pre-training corpora, which restricts scaling and cross-modal knowledge transfer.
Approach
Construct speech-text interleaved pre-training data directly from text corpora by span-corrupting documents and replacing text spans with speech tokens produced by a text-to-token LLM (no actual speech synthesis), paired with a supervised ASR-derived single-codebook 12.5Hz speech tokenizer.
Key results
- Spoken QA S->T on Llama Questions 64.7 (9B) vs 62.3 for Moshi; Web Questions S 15.9 vs 9.2 for Moshi, with the paper citing average spoken QA gains from 13% to 31%.
- 12.5Hz tokenizer reaches WER 8.43 and MOSNet 3.39 on LibriSpeech at 175 bps bitrate.
- Removing synthetic interleaved data drops Llama Questions S->T from 64.7 to 2.3 in the 9B ablation; general QA chatbot score 3.69 vs 2.42 for Moshi in text-guided mode.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale synthetic interleaved pre-training data from 200B to 600B tokens (ablation) | 9B SpeechLM · Llama Questions (S) | accuracy: 44.0 -> 50.7 | Helped |
| Scale synthetic interleaved pre-training data from 200B to 600B tokens (ablation) | 9B SpeechLM · Web Questions (S) | accuracy: 13.3 -> 15.9 | Helped |
| Remove synthetic interleaved data (ablation) | 9B SpeechLM · Llama Questions (S) | accuracy: 2.3 -> 50.7 | Helped |
| Remove synthetic interleaved data (ablation) | 1.5B SpeechLM · sTopic-StoryCloze (T→S) | accuracy: 55.3 -> 81.4 | Helped |
| Remove unsupervised speech data (ablation) | 1.5B SpeechLM · Llama Questions (S→T) | accuracy: 42.7 -> 38.3 | Hurt |
| Remove text data (ablation) | 1.5B SpeechLM · Web Questions (S) | accuracy: 5.4 -> 6.3 | Helped |
| Remove ASR and TTS data (ablation) | 1.5B SpeechLM · sStoryCloze (S→T) | accuracy: 64.0 -> 68.5 | Helped |
| Remove pre-training (ablation) | 9B text-guided spoken chatbot · General QA | score: 3.69 -> 2.20 | Hurt |
| Remove pre-training (ablation) | 9B text-guided spoken chatbot · Knowledge | score: 4.70 -> 0.90 | Hurt |
| Remove interleaved pre-training (ablation) | 9B text-guided spoken chatbot · Knowledge | score: 4.70 -> 1.00 | Hurt |
| Reduce speech tokenizer frame rate from 12.5Hz to 6.25Hz (ablation) | Ours speech tokenizer · LibriSpeech | WER: 8.43 -> 14.41 | Hurt |
| Reduce speech tokenizer frame rate from 50Hz to 12.5Hz (ablation) | Ours speech tokenizer · LibriSpeech | WER: 6.24 -> 8.43 | Hurt |
Surprising findings
- Without any interleaved data the model performs near zero on spoken QA, showing interleaving is essential for knowledge transfer.
- Removing text or speech-text parallel data slightly improves many 1.5B benchmarks, attributed to modality capacity competition in small models.
Limitations
- Synthetic interleaved speech quality is lower for text spans from web data (text-to-token WER 11.62% English, 9.34% Chinese vs 3.20% on VCTK).
- Some baselines (e.g., Moshi, Spirit-LM) use different or private training data, complicating comparison; Moshi trained on 7M hours of speech versus this work's 700k hours.
- Evaluation of chatbots relies on GPT-4 judging and Whisper transcription, and the 12.5Hz tokenizer shows higher WER than some other tokenizers.
Experimental setup
Datasets: FineWeb-Edu Chinese-Fineweb-Edu Emilia LibriSpeech GigaSpeech MLS-Eng WenetSpeech CommonVoice AISHELL-1 Magpie Yodas2 Libri-Light
Benchmarks: Spoken StoryCloze Spoken TopicStoryCloze Web Questions Llama Questions TriviaQA LibriSpeech AlpacaEval VCTK
Metrics: accuracy WER VisQOL MOSNet UTMOS GPT-4 score knowledge accuracy
Baselines: GSLM AudioLM TWIST Spirit-LM SpeechGPT Spectron Moshi Llama-Omni Mini-Omni
Builds on: Spirit-LM CosyVoice Whisper TWIST Moshi
Closest papers by meaning
- Speech Self-Supervised Learning Using Diffusion Model Synthetic Data (ICML 2024 · similarity 0.88)
- Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation (NeurIPS 2024 · similarity 0.88)
- TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling (NeurIPS 2025 · similarity 0.87)
- SyllableLM: Learning Coarse Semantic Units for Speech Language Models (ICLR 2025 · similarity 0.87)
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.87)
- IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTS (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.