Research map / Graph learning / Graph self-supervised learning
SECodec: Structural Entropy-based Compressive Speech Representation Codec for Speech Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SECodec, a speech codec whose codebook is built and whose quantization is done by minimizing 2D structural entropy on a speech feature graph, plus SESLM, a VALL-E-style speech LM built on it.
Problem
Speech discretization for speech language models relies on a predefined codebook size and Euclidean-distance quantization, which causes information loss, audio distortion, and overly long tokens that are hard to train.
Approach
Model speech features as a graph and use hierarchical, disentangled 2D structural entropy minimization to determine the codebook size automatically, then quantize each new frame by selecting the cluster that minimizes the change in 2D structural entropy instead of nearest Euclidean neighbor.
Key results
- SECodec (468 codebook entries) achieves WER 4.63 and ViSQOL 4.40 versus EnCodec-1024 at 5.01 and 4.37 on LibriSpeech reconstruction, with 19.19M parameters.
- SESLM achieves WER 4.97 and SIM-o 0.634 on VCTK zero-shot TTS, versus VALL-E at 7.09 and 0.501.
- Removing SE quantization raises WER from 4.63 to 4.71 and lowers MUSHRA from 90.79 to 87.62.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SECodec instead of EnCodec (K=1024) | speech reconstruction · LibriSpeech test | WER: 5.01 -> 4.63 | Helped |
| use SECodec instead of EnCodec (K=1024) | speech reconstruction · LibriSpeech test | MCD: 5.89 -> 5.12 | Helped |
| use SECodec instead of EnCodec (K=1024) | speech reconstruction · LibriSpeech test | RMSE: 32.59 -> 31.97 | Helped |
| use SECodec instead of EnCodec (K=1024) | speech reconstruction · LibriSpeech test | ViSQOL: 4.37 -> 4.40 | Helped |
| use SESLM instead of USLM | zero-shot TTS · VCTK | WER: 5.79 -> 4.97 | Helped |
| use SESLM instead of USLM | zero-shot TTS · VCTK | SIM-o: 0.602 -> 0.634 | Helped |
| use SESLM instead of USLM | zero-shot TTS · VCTK | QMOS: 3.50($\pm0.13$) -> 3.63($\pm0.12$) | Helped |
| use 100000 graph nodes instead of 10000 (ablation) | SECodec · LibriSpeech test | WER: 4.63 -> 4.91 | Hurt |
| use 50000 graph nodes instead of 10000 (ablation) | SECodec · LibriSpeech test | WER: 4.63 -> 4.82 | Hurt |
| use edge threshold >0.5 instead of >0.2 (ablation) | SECodec · LibriSpeech test | WER: 4.63 -> 4.87 | Hurt |
| add the SE codebook (ablation) | SECodec · LibriSpeech test | WER: 4.89 -> 4.63 | Helped |
| add SE quantization (ablation) | SECodec · LibriSpeech test | MUSHRA: 87.62 -> 90.79 | Helped |
Surprising findings
- Scaling EnCodec's codebook to 4096 entries causes the model to fail to converge (WER 91.23).
- A smaller SE-selected codebook (468 entries) gives better reconstruction than larger codebooks, suggesting codebook size matters more than capacity.
Limitations
- Vanilla 2D SE minimization has O(|V|^3) complexity, motivating the hierarchical approximation.
- Evaluation of TTS is limited to VCTK with a single 3-second prompt setup.
- The paper reports results across only a few node and edge configurations.
Experimental setup
Datasets: LibriSpeech Multilingual LibriSpeech (English subset) VCTK
Benchmarks: LibriSpeech test set VCTK
Metrics: WER MCD RMSE ViSQOL MUSHRA SIM-o SIM-r QMOS SMOS
Baselines: EnCodec SpeechTokenizer VALL-E USLM
Builds on: RVQ-GANs EnCodec VALL-E vCLUB FunCodec
Closest papers by meaning
- CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR 2024 · similarity 0.86)
- Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model (AAAI 2025 · similarity 0.86)
- Structural Entropy Guided Probabilistic Coding (AAAI 2025 · similarity 0.86)
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.85)
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.85)
- Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.