Research map / Architectures and efficiency / Attention and transformers
CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLaM-TTS, a zero-shot TTS system that compresses mel-spectrograms with a probabilistic residual VQ autoencoder and uses a ByT5-based latent language model to predict all depth tokens of a frame in one autoregressive step.
Problem
Neural codec language models for zero-shot TTS are slow and hard to scale because audio token sequences are long and require multiple cascaded stages to model multiple token streams.
Approach
Mean-field variational inference yields a probabilistic RVQ (Mel-VAE) that compresses mel-spectrograms to 10Hz codes with 32 depths; a latent LM with a Gaussian mixture head predicts continuous latents that are quantized into all depth tokens at once, removing the coarse-to-fine second stage.
Key results
- On LibriSpeech continuation, CLaM-en reaches WER 2.36 and SIM-o 0.4767, ranking first or second among compared systems except VoiceBox, which is stronger on quality metrics.
- Mel-VAE with the proposed probabilistic RVQ beats Encodec at a comparable bitrate on reconstruction (PESQ 2.95 vs 2.59; ViSQOL 4.66 vs 4.26).
- CLaM-en generates 10s speech in 4.15s, faster than the Vall-E generation speed reported in its paper and faster than VoiceBox with 64 decoding steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed probabilistic RVQ instead of conventional RVQ (ablation) | Mel-VAE speech codec · Speech reconstruction | PESQ: 2.54 -> 2.63 | Helped |
| Use the proposed probabilistic RVQ instead of conventional RVQ (ablation) | Mel-VAE speech codec · Speech reconstruction | ViSQOL: 4.44 -> 4.48 | Helped |
| Use the proposed RVQ codec instead of Encodec (ablation) | Mel-VAE speech codec · Speech reconstruction | PESQ: 2.59 -> 2.95 | Helped |
| Use the proposed RVQ codec instead of Encodec (ablation) | Mel-VAE speech codec · Speech reconstruction | ViSQOL: 4.26 -> 4.66 | Helped |
| Use word-based rather than duration-based prompting (ablation) | CLaM-en · LibriSpeech test-clean cross-sentence | WER: 8.68 -> 5.11 | Helped |
| Use word-based rather than duration-based prompting (ablation) | CLaM-en · LibriSpeech test-clean cross-sentence | CER: 5.69 -> 2.87 | Helped |
| Use byte-sequence input rather than T5 subword input (ablation) | Text-to-Code model · LibriSpeech test-clean continuation | WER: 9.48 -> 2.79 | Helped |
| Use phoneme input rather than byte-sequence input (ablation) | Text-to-Code model · LibriSpeech test-clean continuation | WER: 2.79 -> 2.62 | Helped |
| Reduce codeword rate from default df8 to df16 (ablation) | Mel-VAE · Speech reconstruction | PESQ: 2.95 -> 2.42 | Hurt |
| Increase codeword rate from default df8 to df4 (ablation) | Mel-VAE · Speech reconstruction | PESQ: 2.95 -> 3.10 | Helped |
| Increase codeword emitting frequency (ablation) | CLaM-TTS latent language model · Speech generation | speaker similarity | Helped |
| Increase codeword emitting frequency (ablation) | CLaM-TTS latent language model · Speech generation | intelligibility | Mixed |
Surprising findings
- VoiceBox, a phoneme-and-duration model, outperforms CLaM-en on most English metrics, though CLaM-en needs no phonemes or durations.
- Among T5 variants, a phoneme encoder slightly beats ByT5 on continuation WER and SIM-o.
Limitations
- Some words are muddled, omitted, or repeated, attributed to autoregressive modeling.
- Training data is mostly audiobook reading style, limiting expressiveness and speaking-style diversity.
- Codeword rate of 10Hz was set from English phoneme rate and may need adjustment for other languages or speakers.
Experimental setup
Datasets: MLS GigaSpeech LibriTTS-R VCTK LJSpeech AIHub 14 AIHub 15 KsponSpeech WeNetSpeech ReazonSpeech KokoroSpeech LibriSpeech test-clean
Benchmarks: LibriSpeech test-clean MLS multilingual subsets
Metrics: WER CER SIM-o SIM-r QMOS SMOS CMOS PESQ ViSQOL inference time
Baselines: YourTTS Vall-E SPEAR-TTS VoiceBox Encodec-based RVQ
Builds on: RQ-VAE VALL-E ByT5 BigVGAN Encodec
Closest papers by meaning
- ELLA-V: Stable Neural Codec Language Modeling with Alignment-Guided Sequence Reordering (AAAI 2025 · similarity 0.91)
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.90)
- MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer (ICLR 2025 · similarity 0.89)
- HALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis (ICLR 2025 · similarity 0.89)
- TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling (NeurIPS 2025 · similarity 0.89)
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.