Research map / Large language models / Language, safety and interpretability
ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ALMTokenizer, a low-bitrate audio codec that uses learnable query tokens with transformer compression, semantic priors, MAE and AR losses, and two-stage training to produce semantically rich discrete audio tokens for audio language models.
Problem
Neural audio codecs such as Encodec produce high-bitrate, semantically weak tokens; reducing frame rate by larger strides degrades reconstruction, which hinders audio language modeling.
Approach
Replaces uniform frame-level down-sampling with learnable query tokens that aggregate contextual information across audio frames via a transformer, combined with fixed semantic-prior VQ codebooks (from k-means on self-supervised embeddings), MAE and autoregressive losses, and two-stage training.
Key results
- Ablation (Table 6) shows query-based framework, MAE and AR losses, and two-stage training each improve semantic performance relative to a MimiCodec-style baseline.
- Mask rate 20-30% and loss weights (lambda1=0.5, lambda2=0.1) were chosen as best trade-off between semantics and reconstruction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the query-based compression framework instead of the reproduced MimiCodec-style baseline (ablation) | ALMTokenizer · VCTK | audio compression performance | Helped |
| Use the query-based compression framework instead of the reproduced MimiCodec-style baseline (ablation) | ALMTokenizer · VCTK | semantic performance | Helped |
| Add MAE loss (ablation) | ALMTokenizer · VCTK | semantic performance | Helped |
| Add autoregressive language-model loss (ablation) | ALMTokenizer · VCTK | semantic performance | Helped |
| Use two-stage training instead of one-stage training (ablation) | ALMTokenizer · VCTK | reconstruction performance | Helped |
| Use two-stage training instead of one-stage training (ablation) | ALMTokenizer · VCTK | semantic performance | Helped |
| Add semantic VQ priors (ablation) | ALMTokenizer · semantic tasks | semantic performance | Helped |
| Add semantic VQ priors (ablation) | ALMTokenizer · audio reconstruction | reconstruction quality | Hurt |
| Add autoregressive loss (ablation) | ALMTokenizer · audio reconstruction | reconstruction quality | Hurt |
| Increase the MAE masking rate to 30–40% from the tested lower ranges (ablation) | ALMTokenizer · audio reconstruction | reconstruction quality | Hurt |
| Increase the MAE masking rate to 30–40% from the tested lower ranges (ablation) | ALMTokenizer · semantic representation | semantic performance | Helped |
| Set MAE and AR loss weights to (0.5, 0.1) instead of (1, 1) or (0.5, 0.5) (ablation) | ALMTokenizer · VCTK | performance | Helped |
Surprising findings
- Semantic VQ priors give only marginal semantic gains while slightly degrading reconstruction quality.
- Higher MAE mask rates (30-40%) improve semantics but harm reconstruction.
Limitations
- Trade-off: AR loss and semantic VQ priors may slightly hurt reconstruction metrics.
- Codebooks are fixed during training, unlike traditional VQ training, which may limit joint optimization.
- Variable bitrate applicability to downstream language modeling is unexplored.
Experimental setup
Datasets: VCTK LibriTTS
Benchmarks: VCTK LibriTTS
Metrics: reconstruction quality semantic performance MUSHRA
Baselines: Encodec DAC MimiCodec SpeechTokenizer
Builds on: Encodec SoundStream MimiCodec SpeechTokenizer MAE
Closest papers by meaning
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.94)
- Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model (AAAI 2025 · similarity 0.92)
- CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR 2024 · similarity 0.90)
- A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization (NeurIPS 2026 · similarity 0.90)
- Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.89)
- OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.