Research map / Architectures and efficiency / Attention and transformers
Scaling Transformers for Low-Bitrate High-Quality Speech Coding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scales a large transformer audio autoencoder (TAAE, ~950M params) with an FSQ bottleneck and post-hoc residual FSQ to deliver state-of-the-art 16 kHz speech coding at 400 and 700 bps.
Problem
Neural speech codecs for generative pipelines need extremely low bitrates with high quality, but existing codecs use small convolutional or recurrent architectures and RVQ bottlenecks that suffer from codebook utilization issues and hierarchical token dependencies.
Approach
A predominantly transformer-based encoder/decoder (Transformer Audio AutoEncoder) scaled to about 1B parameters, combined with a modified Finite Scalar Quantization bottleneck trained with uniform-noise/straight-through mixing, and a post-hoc decomposition of FSQ into residual levels that allows bitrate changes without retraining.
Key results
- At 400 bps and 700 bps, TAAE reaches PESQ 2.96 and 3.09 and STOI 0.90 and 0.92, outperforming baselines such as Mimi at 1100 bps (PESQ 3.01, STOI 0.90) on several metrics at far lower bitrate.
- A MUSHRA listening test with 24 participants found TAAE significantly outperforms recent speech codecs and is close to the ground truth.
- Scaling from 240M to 950M parameters improves 700 bps SI-SDR from 3.52 to 4.80 and PESQ from 2.74 to 2.82.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TAAE at 400 bps instead of the reported higher-bitrate Mimi baseline | Speech codec · LibriSpeech test-clean | SI-SDR: 2.20 -> 3.18 | Helped |
| Use TAAE at 700 bps instead of Mimi at 1100 bps | Speech codec · LibriSpeech test-clean | PESQ: 3.01 -> 3.09 | Helped |
| Add WavLM perceptual feature-matching loss during finetuning (ablation) | TAAE · LibriSpeech test-clean, 700 bps | PESQ: 2.82 -> 3.09 | Helped |
| Add WavLM perceptual feature-matching loss during finetuning (ablation) | TAAE · LibriSpeech test-clean, 700 bps | SI-SDR: 4.80 -> 4.73 | Hurt |
| Increase model size from 240M to 950M parameters (ablation) | TAAE · LibriSpeech test-clean, 700 bps | SI-SDR: 3.52 -> 4.80 | Helped |
| Lower latent rate to 12.5 Hz and increase FSQ dimension to 8 (ablation) | TAAE · LibriSpeech test-clean, 200–650 bps | SI-SDR: 3.18 -> -1.40; 4.73 -> 3.37 | Mixed |
| Make the attention window and convolution layers causal (ablation) | TAAE · LibriSpeech test-clean, 700 bps | SI-SDR: 4.73 -> 4.04 | Hurt |
| Remove FSQ quantization and use continuous latents (ablation) | TAAE · LibriSpeech test-clean | SI-SDR: 4.73 -> 5.08 | Helped |
| Use two residual FSQ tokens rather than one token (ablation) | TAAE · LibriSpeech test-clean, 400 vs 700 bps | SI-SDR: 3.18 -> 4.73 | Helped |
Surprising findings
- Removing the quantizer (continuous latents) gives only modestly better objective scores (PESQ 3.12 vs 3.09 at 700 bps), suggesting the FSQ bottleneck costs little quality.
Limitations
- Trained only on about 100k hours of English, 16 kHz audiobook-dominated speech, limiting generalization to other settings and sampling rates.
- Large parameter count implies higher computational cost than the baselines.
- Causal streaming variant shows some degradation and was only obtained by finetuning.
Experimental setup
Datasets: Librilight Multilingual LibriSpeech (English) LibriSpeech test-clean MLS (multilingual test subsets) LibriSpeech train-clean-360
Benchmarks: LibriSpeech test-clean Multilingual LibriSpeech
Metrics: SI-SDR Mel distance STFT distance PESQ STOI MOSNet MUSHRA normalized entropy Huffman-coded bitrate real-time factor
Baselines: DAC Encodec SpeechTokenizer SemantiCodec Mimi unitHiFi-GAN
Builds on: Finite Scalar Quantization Encodec DAC WavLM FlashAttention
Closest papers by meaning
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.88)
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.88)
- CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR 2024 · similarity 0.87)
- FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks (NeurIPS 2025 · similarity 0.87)
- TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling (NeurIPS 2025 · similarity 0.86)
- Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.