Research map / Large language models / Language, safety and interpretability
SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpeechTokenizer, an RVQ-based speech tokenizer whose first quantizer is guided by HuBERT semantic distillation, and a unified speech LM (USLM) built on it, also introducing the SLMTokBench token benchmark.
Problem
Existing discrete speech tokens (semantic tokens from SSL models and acoustic tokens from neural codecs) are not designed for speech language modeling; semantic tokens lose timbre while acoustic tokens lack text alignment, and hierarchical multi-stage SpeechLMs are complex.
Approach
A single RVQ-based encoder-decoder codec whose first quantizer is distilled from HuBERT (continuous representation with a per-dimension cosine loss plus pseudo-label prediction), so the first layer carries content and subsequent layers carry paralinguistic information; a unified AR+NAR speech LM is built on these tokens.
Key results
- SpeechTokenizer RVQ-1:8 achieves WER 5.04 and ViSQOL 4.30 on reconstruction versus EnCodec's 5.11 and 4.37, with MUSHRA 90.55 vs 79.86 for EnCodec.
- In zero-shot TTS on VCTK, USLM reaches WER 6.5 and SIM 0.84 versus VALL-E's 7.9 and 0.75, with MOS 3.63 vs 3.08.
- SpeechTokenizer RVQ-1 reaches mutual information 32.9 (HuBERT L9 teacher) compared with 16.5 for EnCodec RVQ-1, and PNMI 0.71 vs 0.28.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SpeechTokenizer rather than EnCodec for speech reconstruction | LibriSpeech test reconstruction · LibriSpeech test | WER: 5.11 -> 5.04 | Helped |
| use SpeechTokenizer rather than EnCodec for speech reconstruction | LibriSpeech test reconstruction · LibriSpeech test | ViSQOL: 4.37 -> 4.30 | Hurt |
| use SpeechTokenizer rather than EnCodec for speech reconstruction | LibriSpeech test reconstruction · LibriSpeech test | MUSHRA: 79.86 -> 90.55 | Helped |
| use USLM rather than VALL-E | zero-shot TTS · VCTK | WER: 7.9 -> 6.5 | Helped |
| use USLM rather than VALL-E | zero-shot TTS · VCTK | SIM: 0.75 -> 0.84 | Helped |
| use USLM rather than VALL-E | zero-shot TTS · VCTK | MOS: 3.08 -> 3.63 | Helped |
| use USLM rather than VALL-E | zero-shot TTS · VCTK | SMOS: 3.31 -> 3.45 | Helped |
| use HuBERT L9 rather than HuBERT avg as the semantic teacher (ablation) | SpeechTokenizer RVQ-1 · SLMTokBench | MI: 30.9 -> 32.9 | Helped |
| use HuBERT L9 rather than HuBERT avg as the semantic teacher (ablation) | SpeechTokenizer RVQ-1:8 · SLMTokBench resynthesized speech | WER: 5.04 -> 5.31 | Hurt |
| use HuBERT units rather than HuBERT avg as the semantic teacher (ablation) | SpeechTokenizer · SLMTokBench | MI: 30.9 -> 24.2; 29.7 -> 25.1 | Hurt |
| replace BiLSTM with LSTM in the encoder (ablation) | SpeechTokenizer · SLMTokBench RVQ-1 text alignment | MI: 30.9 -> 27.60 | Hurt |
| replace D-Axis with T-Axis continuous distillation loss (ablation) | SpeechTokenizer · SLMTokBench RVQ-1 text alignment | MI: 30.9 -> 26.65 | Hurt |
Surprising findings
- Using HuBERT L9 continuous representations as teacher outperforms discrete HuBERT units as teacher on text alignment and preservation metrics.
- The first-layer RVQ tokens alone lose timbre and prosody, yet speaker info is recovered from layers 2-8, which enables one-shot voice conversion by swapping RVQ-1 with reference RVQ-2:8 tokens.
Limitations
- SpeechTokenizer RVQ-1 alone has low speaker similarity and poor reconstruction (WER 15.58 on downstream), so the full RVQ stack is needed for quality.
- Trained only on English data; unseen-language results are qualitative (demo samples and spectrograms) rather than quantitatively evaluated.
- Subjective MOS/SMOS evaluations use a small number of raters (12 and 6).
Experimental setup
Datasets: LibriSpeech Multilingual LibriSpeech (English subset) VCTK TIMIT Aishell-3
Benchmarks: SLMTokBench LibriSpeech test-clean (reconstruction) VCTK (zero-shot TTS)
Metrics: mutual information WER speaker similarity ViSQOL MUSHRA MOS SMOS PNMI
Baselines: EnCodec VALL-E HuBERT units (KM500) SPEAR-TTS
Builds on: EnCodec SoundStream HuBERT VALL-E HiFi-Codec
Closest papers by meaning
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.89)
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.88)
- TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling (NeurIPS 2025 · similarity 0.87)
- CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR 2024 · similarity 0.87)
- UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (AAAI 2024 · similarity 0.86)
- Model-Aware Tokenizer Transfer (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.