Research map / Label-efficient and robust learning / Self-supervised and contrastive

FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks

NeurIPS 2025 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Self-supervised speech and brain decoding · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces FocalCodec, a single-binary-codebook hybrid speech codec built on focal modulation and WavLM-large layer-6 features, operating at 0.16–0.65 kbps.

Problem

Low-bitrate speech tokenization that keeps both semantic and acoustic information without multi-codebook designs, distillation, or supervised fine-tuning.

Approach

A compressor-quantizer-decompressor between a frozen WavLM-large (first 6 layers) encoder and a Vocos decoder, where the compressor uses focal modulation blocks and quantization uses binary spherical quantization (BSQ) with a 2^13 implicit codebook; the codec is trained in two decoupled stages.

Key results

• FocalCodec@50 (0.65 kbps) achieves the lowest dWER on LibriSpeech test-clean (2.18) versus BigCodec's 2.55 at 1.04 kbps.
• On noisy Libri1Mix resynthesis, FocalCodec@50 reaches dWER 27.89 vs. 53.26 for BigCodec.
• FocalCodec@12.5 operates at 0.16 kbps and remains competitive with EnCodec at 1.50 kbps on clean resynthesis (UTMOS 4.22, dWER 7.94).

Measured effects reported in the paper

Surprising findings

• FocalCodec's UTMOS tends to increase at lower bitrates, attributed to smoothing from downsampling, which the authors note makes UTMOS an incomplete quality signal.
• Acoustic codecs such as BigCodec attempt to reconstruct background noise, while semantic-leaning FocalCodec performs implicit denoising.

Limitations

• Trained on only a few hundred hours of clean English speech, so it is undertrained relative to baselines trained on far larger corpora.
• Non-causal; streaming at 500 ms chunks degrades dWER, and low-latency causal version required further changes.
• Downstream generative tasks (SS, TTS) remain far from practical utility; TTS not state-of-the-art.

Experimental setup

Datasets: LibriTTS LibriSpeech Multilingual LibriSpeech VoiceBank Libri1Mix LibriMix WHAM! VCTK IEMOCAP Libri2Mix

Benchmarks: LibriSpeech test-clean Multilingual LibriSpeech 700 VoiceBank Libri1Mix VCTK LibriSpeech ASR IEMOCAP VoiceBank SE Libri2Mix LibriSpeech TTS

Metrics: UTMOS DNSMOS dWER Sim code usage normalized entropy RTF WER ER MUSHRA

Baselines: EnCodec DAC WavLM6-KM SpeechTokenizer SemantiCodec Mimi WavTokenizer BigCodec Stable Codec TS3-Codec

Builds on: VQ-VAE focal modulation binary spherical quantization Vocos HiFi-GAN

Closest papers by meaning

• Scaling Transformers for Low-Bitrate High-Quality Speech Coding (ICLR 2025 · similarity 0.87)
• ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.86)
• Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model (AAAI 2025 · similarity 0.86)
• CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR 2024 · similarity 0.86)
• Time-Masked Transformers with Lightweight Test-Time Adaptation for Neural Speech Decoding (NeurIPS 2025 · similarity 0.85)
• SECodec: Structural Entropy-based Compressive Speech Representation Codec for Speech Language Models (AAAI 2025 · similarity 0.85)