Research map / Label-efficient and robust learning / Self-supervised and contrastive
Sylber: Syllabic Embedding Representation of Speech from Raw Audio
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Sylber learns syllable-level speech embeddings by iteratively distilling from its own unsupervised syllable segmentation, enabling a linear-time segmenter, low-rate syllabic tokens (4.27 Tok/s), and emergent categorical perception.
Problem
Current neural speech representations lack syllabic structure and yield dense, high-frequency token sequences that are costly for spoken language modeling and resynthesis; existing syllable induction methods give noisy boundaries and quadratic-time segmentation.
Approach
Self-segmentation distillation: initialize from SDHuBERT, regress frame features onto segment-averaged teacher embeddings from unsupervised segments, then update the teacher with the stage-1 student and segment online with a linear-time greedy algorithm, producing flat, clean syllabic embeddings that are naturally categorical.
Key results
- Sylber achieves syllable detection F1 72.2 and R-value 75.9 on LibriSpeech, outperforming SDHuBERT (67.5 F1) with O(n) segmentation.
- Sylber tokens average 4.27 Tok/s, about 6-7x fewer than HuBERT units, with WER 4.88 unquantized and 7.95 at 20K clusters.
- Sylber-uLM trained on 66K hours of LibriLight reaches sWUGGY 76.31 and sBLIMP 60.54 at 125M parameters, with sBLIMP above TWIST-13B (59.20).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use self-segmentation distillation instead of the previous [cite] approach | syllable detection and discovery on LibriSpeech test · LibriSpeech test | F1: 70.3 -> 72.2 | Helped |
| use Sylber rather than the previous [cite] approach | syllable discovery on LibriSpeech test · LibriSpeech test | syllabic purity: 59.4 -> 64.0 | Helped |
| use Sylber greedy segmentation instead of MinCut (ablation) | Sylber syllable features · LibriSpeech test | F1: 72.2 -> 72.2 | No clear effect |
| use linear-time greedy segmentation on SDHuBERT features (ablation) | SDHuBERT · LibriSpeech test | F1: 67.5 -> 61.2 | Hurt |
| use linear-time greedy segmentation on HuBERT features (ablation) | HuBERT · LibriSpeech test | F1: 39.0 -> 42.8 | Hurt |
| add second-stage training with updated teacher segments (ablation) | Sylber · LibriSpeech test | F1: 71.4 -> 72.2 | Helped |
| add the denoising objective (ablation) | Sylber · noisy audio | robustness | Helped |
| increase token vocabulary granularity from 5K to 20K (ablation) | Sylber token-to-speech model · LibriTTS-R test-clean | WER: 8.70 -> 7.95 | Helped |
| use Sylber units instead of SDHuBERT tokens | token-to-speech resynthesis · LibriTTS-R test-clean, 20K vocab | WER: 8.63 -> 7.95 | Helped |
| use Sylber units instead of HuBERT 2K units | token-to-speech resynthesis · LibriTTS-R test-clean | tokens/second: 33.62 -> 4.27 | Helped |
| use Sylber instead of SDHuBERT tokens | coding efficiency at 5K vocabulary · LibriTTS-R test-clean | bitrate: 64.39 -> 52.43 | Helped |
| add a silence token when the gap between tokens exceeds 140 ms (ablation) | Sylber-uLM · LibriLight, 66K hours, 20K vocab | sWUGGY: 76.31 -> 78.03 | Helped |
Surprising findings
- Sylber's embedding space shows categorical perception (lowest discriminability index, 0.112) even though the training loss involves no categorical objective.
- Sylber segmentation generalizes to conversational English, Spanish, and Mandarin without tuning, trained only on English audiobooks.
Limitations
- Model is designed as a coding framework and is not suitable as a universal speech representation; it degrades on some SUPERB downstream tasks.
- Quantized resynthesis is less articulatory-accurate than SDHuBERT; the best intelligibility requires unquantized features.
- Several thresholds (merge threshold, shifts) were set heuristically after visual inspection.
Experimental setup
Datasets: LibriSpeech LibriLight LibriTTS-R Fisher MLS AISHELL-3
Benchmarks: LibriSpeech syllable segmentation Fisher MLS AISHELL-3 LibriTTS-R test-clean resynthesis sWUGGY sBLIMP SUPERB
Metrics: precision recall F1 R-value syllable purity cluster purity mutual information WER CER UTMOS
Baselines: HuBERT VGHuBERT SDHuBERT GSLM tGSLM NAST TWIST Wav2Vec2 WavLM MinCut
Builds on: SDHuBERT HuBERT GSLM SPARC MinCut
Closest papers by meaning
- SyllableLM: Learning Coarse Semantic Units for Speech Language Models (ICLR 2025 · similarity 0.89)
- OWLS: Scaling Laws for Multilingual Speech Recognition and Translation Models (ICML 2025 · similarity 0.84)
- SECodec: Structural Entropy-based Compressive Speech Representation Codec for Speech Language Models (AAAI 2025 · similarity 0.84)
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.84)
- CA-SSLR: Condition-Aware Self-Supervised Learning Representation for Generalized Speech Processing (NeurIPS 2024 · similarity 0.84)
- Long-Form Speech Generation with Spoken Language Models (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.