Research map / Large language models / Language, safety and interpretability
Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes X-Codec, an audio codec that concatenates self-supervised semantic features (HuBERT) with acoustic features before RVQ and adds a semantic reconstruction loss, improving audio language models for TTS, music continuation, and text-to-sound.
Problem
Acoustic codecs such as EnCodec and DAC are designed for compression, not audio language modeling; they lack phonetic/semantic discriminability, causing high WER and content errors in LLM-based audio generation like VALL-E.
Approach
Inject semantic features from a pre-trained self-supervised encoder (HuBERT or wav2vec 2.0) into the codec by concatenating them with acoustic features before RVQ, and reconstruct the semantic features after quantization with an MSE loss, yielding a uniform token stream without disentangling semantic and acoustic tokens.
Key results
- On zero-shot continuation TTS with VALL-E (AR+NAR), X-Codec-wavlm-base-plus reduces WER to 3.26 from 7.70 for the baseline acoustic codec and 5.24 for SpeechTokenizer, with UTMOS 4.22.
- On ABX phonetic discriminability with 8 quantizers, X-Codec achieves 3.3 within and 4.3 across error rates versus 20.1/28.3 for the baseline acoustic codec.
- On text-to-sound generation, X-Codec improves FD from 59.03 to 46.31 and FAD from 6.19 to 4.10 over the baseline acoustic codec.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Integrate semantic information into the codec | VALL-E with X-Codec-hubert vs. Baseline Acoustic Codec · LibriSpeech-test-clean zero-shot TTS, AR+NAR | WER: 7.70 -> 4.07 | Helped |
| Integrate semantic information into the codec | VALL-E with X-Codec-hubert vs. SpeechTokenizer · LibriSpeech-test-clean zero-shot TTS, AR+NAR | Sim-O: 0.36 -> 0.42 | Helped |
| Use X-Codec-wavlm-base-plus instead of X-Codec-hubert | VALL-E · LibriSpeech-test-clean zero-shot TTS, AR+NAR | WER: 4.07 -> 3.26 | Helped |
| Integrate semantic information into the codec | GPT-2-medium music continuation with X-Codec vs. Acoustic codec · MUSDB18 music continuation | FD: 16.17 -> 12.66 | Helped |
| Integrate semantic information into the codec | GPT-2-medium text-to-sound with X-Codec vs. Acoustic codec · Text-to-sound | CLAP: 0.417 -> 0.483 | Helped |
| Increase X-Codec quantization layers from n_q=1 to n_q=8 (ablation) | X-Codec · Phonetic ABX within error rate | ABX error rate: 3.9 -> 3.3 | Helped |
| Increase SpeechTokenizer quantization layers from n_q=1 to n_q=8 (ablation) | SpeechTokenizer · Phonetic ABX within error rate | ABX error rate: 3.5 -> 3.6 | Hurt |
| Integrate semantic information into the codec | X-codec vs. Baseline, n_q=1 · LibriTTS reconstruction | Mel DT.: 0.79 -> 0.86 | Hurt |
| Integrate semantic information into the codec | X-codec vs. Baseline, n_q=8 · LibriTTS reconstruction | UTMOS: 3.72 -> 4.01 | Helped |
| Use four flattened VQs instead of the first VQ (ablation) | X-codec music continuation · MUSDB18 music continuation | FAD: 7.08 -> 0.20 | Helped |
Surprising findings
- Reconstructing the semantic features slightly impairs Mel and STFT distance metrics, a trade-off counterbalanced by UTMOS gains.
- SpeechTokenizer shows lower speaker similarity (Sim-O) than the baseline acoustic codec, attributed to its content-focused first stage.
Limitations
- Semantic module adds computational overhead, which may slow compression and transmission.
- Slight degradation in Mel and STFT distance reconstruction metrics.
- Music and sound experiments rely on large internal datasets and only compare against the own baseline codec.
Experimental setup
Datasets: LibriSpeech LibriTTS MLS English MUSDB18 AudioCaps WavCaps AudioSet ARCH benchmark librispeech-test-clean Common Voice DNS Challenge 4 FSD50K
Benchmarks: librispeech-test-clean MUSDB18 AudioCaps ARCH
Metrics: WER SIM-O UTMOS ABX error rate FD FAD FD-MERT-layer-9 IS CLAP score Mel distance
Baselines: EnCodec DAC SpeechTokenizer Baseline Acoustic Codec AudioLM MusicGen
Builds on: VALL-E EnCodec SpeechTokenizer DAC HuBERT
Closest papers by meaning
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.92)
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.91)
- UniAudio 1.5: Large Language Model-Driven Audio Codec is A Few-Shot Audio Task Learner (NeurIPS 2024 · similarity 0.90)
- CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR 2024 · similarity 0.89)
- Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.88)
- A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.