Research map / Large language models / Language, safety and interpretability

WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Speech and audio language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces WavTokenizer, a single-quantizer 24kHz audio codec (40 or 75 tokens/s) that uses an enlarged VQ codebook, k-means init and random awakening, an attention-equipped iFFT-based decoder, and multi-scale discriminators to match multi-quantizer codecs' reconstruction quality.

Problem

Existing neural acoustic codecs use multiple quantizers and high token rates (e.g., DAC's 900 tokens/s), which hurts downstream autoregressive language model generation and efficiency; they also lack rich semantic content for unified audio understanding and generation.

Approach

Compress the RVQ hierarchy into one quantizer by enlarging the codebook (to 4096) with k-means initialization and random codebook reactivation, extend the context window to 3 seconds, add an attention block and a Vocos-style inverse-STFT decoder, and use a multi-scale STFT discriminator.

Key results

• WavTokenizer at 0.9 kbps (75 tokens/s) surpasses DAC at 9 kbps (900 tokens/s) on UTMOS across LibriTTS test-clean, test-other and LJSpeech (4.0486 vs 3.9097 on test-clean).
• With a single quantizer, WavTokenizer (75 tokens/s) outperforms single-quantizer DAC (100 tokens/s) on all reported metrics, e.g., UTMOS 4.0486 vs 1.4940 on LibriTTS test-clean.
• In MUSHRA, WavTokenizer scores 96.1 on LibriTTS test-clean vs DAC-9kbps 92.8, and 92.9 on MusicDB vs 92.6.

Measured effects reported in the paper

Surprising findings

• Expanding the codebook from 1024 to 4096 improved UTMOS by about 0.55, while a 16384 codebook dropped utilization to 27%.
• Mirror-style decoder upsampling severely degraded reconstruction under extreme compression (UTMOS 2.78 vs 4.05).

Limitations

• Current codec encoder lacks ASR-level semantic understanding; semantic gains rely on distillation-free design but remain limited.
• Training used only about 8K hours of public data due to compute limits; the authors plan to scale to hundreds of thousands of hours.
• Downstream generative evaluation is limited to TTS on LibriTTS with a ParlerTTS-based model.

Experimental setup

Datasets: LibriTTS VCTK CommonVoice AudioSet Jamendo MusicDB LJSpeech

Benchmarks: LibriTTS test-clean LibriTTS test-other LJSpeech MusicDB AudioSet eval ARCH

Metrics: UTMOS PESQ STOI V/UV F1 MUSHRA classification accuracy CMOS WER speaker similarity RTF

Baselines: DAC Encodec Vocos SpeechTokenizer HiFi-Codec SemantiCodec

Builds on: SoundStream EnCodec Vocos DAC ParlerTTS

Closest papers by meaning

• ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.94)
• Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model (AAAI 2025 · similarity 0.91)
• A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization (NeurIPS 2026 · similarity 0.90)
• SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models (ICLR 2024 · similarity 0.89)
• Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.89)
• CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR 2024 · similarity 0.89)