Research map / Large language models / Language, safety and interpretability
SyllableLM: Learning Coarse Semantic Units for Speech Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SyllableLM, a speech language model trained on low-rate (5-8.33Hz) self-supervised syllable-like units obtained via LossPred boundary extraction and SylBoost iterative distillation.
Problem
Speech language models rely on high-frequency (25-50 tokens/s) SSL units that are expensive to model and mostly phonetic; coarser semantic units are needed for efficient and semantically strong spoken language modeling.
Approach
Extract noisy syllable-like boundaries from the masked-prediction loss correlations of a pretrained HuBERT (LossPred), then sharpen features with a student-teacher distillation (SylBoost) that pools teacher features over predicted segments, and train an OPT-style LM on the resulting controllable-rate units.
Key results
- SylBoost (Data2Vec2) at 5Hz reaches F1@50ms 73.2 on LibriSpeech test, versus 66.1 for SD-HuBERT.
- Resynthesis WER drops from 37.3 (SD-HuBERT, 60bps) to 7.0 (6.25Hz, 81bps) using SylBoost units.
- SyllableLM 300M (6.25Hz) reaches sBLIMP 63.7 and tSC 75.4, outperforming TWIST 1.3B on sBLIMP and reporting ~30x less GPU hours than AudioLM-scale training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SylBoost with Data2Vec2 rather than SD-HuBERT | syllable boundary detection · LibriSpeech test | F1⁵⁰: 66.1 -> 73.2 | Helped |
| use SylBoost with Data2Vec2 rather than SD-HuBERT | syllable boundary detection · LibriSpeech test | F1²⁰: 32.2 -> 44.6 | Helped |
| bootstrap SylBoost with LossPred boundaries instead of Feat-Sim boundaries (ablation) | SylBoost on HuBERT · LibriSpeech dev-clean boundary detection | F1: 50.4 -> 70.2 | Helped |
| add SylBoost clustering to SD-HuBERT units | unit resynthesis · LibriSpeech test-clean 4–10s resynthesis | WER: 37.3 -> 18.5 | Helped |
| use Data2Vec2 rather than HuBERT as the SylBoost backbone (ablation) | unit resynthesis · LibriSpeech test-clean 4–10s resynthesis | WER: 18.5 -> 12.8 | Helped |
| increase the number of units from 4096 to 16384 (ablation) | SylBoost (Data2Vec2) unit resynthesis · LibriSpeech test-clean 4–10s resynthesis | WER: 12.8 -> 9.1 | Helped |
| tune unit rate and cluster count to 6.25Hz and 8192 units (ablation) | SylBoost (Data2Vec2) unit resynthesis · LibriSpeech test-clean 4–10s resynthesis | WER: 9.1 -> 7.0 | Helped |
| lower unit rate from 8.33Hz to 5.00Hz at 4k units (ablation) | SyllableLM · ZeroSpeech development | sWUGGY: 72.9 -> 65.7 | Hurt |
| lower unit rate from 8.33Hz to 6.25Hz at 4k units (ablation) | SyllableLM · ZeroSpeech development | sBLIMP: 61.8 -> 63.3 | Helped |
| vary unit rate at fixed 4k units (ablation) | SyllableLM · ZeroSpeech development | sWUGGY: 72.9 (8.33Hz), 69.3 (6.25Hz), 65.7 (5.00Hz) | Mixed |
| increase cluster count from 2k to 4k at 8.33Hz (ablation) | SyllableLM · ZeroSpeech development | sWUGGY: 72.1 -> 72.9 | Helped |
| increase cluster count from 4k to 8k at 8.33Hz (ablation) | SyllableLM · ZeroSpeech development | sWUGGY: 72.9 -> 72.9 | No clear effect |
Surprising findings
- Lower unit rates (5Hz) decreased sWUGGY quality, which the authors suspect is due to short dataset audio lengths making tokenization too short.
- BPE on deduplicated units, despite achieving the lowest bitrate outside this work, did not approach the quality of syllable-like units.
Limitations
- LossPred assumes semantic units are separable, one-dimensional and contiguous; it does not handle overlapping sounds or multiple speakers.
- Base encoder quality (Data2Vec2, older and trained on less data) may confound SpeechLM comparisons with TWIST/AudioLM.
- Scaling was limited by compute; the largest SyllableLM is 300M parameters.
Experimental setup
Datasets: LibriSpeech LibriLight ZeroSpeech sWUGGY ZeroSpeech sBLIMP tSC (spoken StoryCloze)
Benchmarks: LibriSpeech syllable boundary detection LibriSpeech resynthesis WER/CER sWUGGY sBLIMP tSC VERT@O-PPX / PPX@O-VERT continuation metrics
Metrics: F1 R-value cluster purity syllable purity WER CER accuracy GPU hours bitrate RTF
Baselines: AudioLM TWIST GSLM Moshi SD-HuBERT BPE speech LM VG-HuBERT
Builds on: HuBERT Data2Vec2 TWIST AudioLM SD-HuBERT
Closest papers by meaning
- Sylber: Syllabic Embedding Representation of Speech from Raw Audio (ICLR 2025 · similarity 0.89)
- Scaling Speech-Text Pre-training with Synthetic Interleaved Data (ICLR 2025 · similarity 0.87)
- TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling (NeurIPS 2025 · similarity 0.86)
- FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing (NeurIPS 2025 · similarity 0.86)
- Long-Form Speech Generation with Spoken Language Models (ICML 2025 · similarity 0.86)
- OWLS: Scaling Laws for Multilingual Speech Recognition and Translation Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.