Research map / Image generation / Autoregressive and latent generation

TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling

NeurIPS 2025 · Image generation › Autoregressive and latent generation · approach: Neural text-to-speech synthesis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TaDiCodec, a single-codebook, single-stage diffusion-autoencoder speech tokenizer at 6.25 Hz that conditions its decoder on text and a speech prompt, and uses it for AR and masked-generative zero-shot TTS.

Problem

Existing speech tokenizers either need multi-layer RVQ or high frame rates, depend on pretrained SSL models for semantic distillation, or require complex two-stage training; they also show a large reconstruction-generation gap for speech language modeling.

Approach

An end-to-end diffusion transformer autoencoder with BSQ quantization (no commitment loss) trained only with a flow-matching loss, where the diffusion decoder is conditioned on the transcript text and a prefix speech prompt, enabling extremely low frame rate tokens.

Key results

• TaDiCodec reaches 6.25 Hz / 0.0875 kbps with a single codebook, achieving WER 2.73 and SIM 0.69 on SeedTTS test-en reconstruction (with decoder continued training).
• TaDiCodec-AR (4B) achieves WER 2.28 on Regular en and 1.19 on Regular zh zero-shot TTS, with WER 9.16 on code-switching en vs. 15.03 for the best baseline.
• TaDiCodec-MGM (0.6B) reaches WER 1.69 on Regular zh with 10 inference steps, and 0.12 RTF.

Measured effects reported in the paper

Surprising findings

• Removing text conditioning from the decoder degrades WER above 10 at 12.5 Hz, indicating text guidance is critical at low token rates.
• Diffusion-based PatchGAN replacement or reduced inference steps degrade quality, with only 5 steps causing a sharp WER rise (7.89).

Limitations

• Diffusion decoding requires multiple inference steps, causing higher latency than GAN-based tokenizers.
• The decoder requires text input, limiting use for pure tokenization without transcripts.
• Evaluated mainly on generation-oriented metrics; signal-level metrics such as PESQ and STOI were not reported.

Experimental setup

Datasets: Emilia SeedTTS test-en SeedTTS test-zh Common Voice

Benchmarks: SeedTTS test-en SeedTTS test-zh Articulatory Code-switching Cross-lingual Common Voice multilingual test sets

Metrics: WER SIM UTMOS CMOS RTF

Baselines: EnCodec DAC SpeechTokenizer Mimi DualCodec BiCodec X-codec 2 WavTokenizer BigCodec TAAE SemantiCodec Vevo Tokenizer

Builds on: flow matching Binary Spherical Quantization Llama Qwen2.5-Instruct MaskGIT

Closest papers by meaning

• CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR 2024 · similarity 0.89)
• DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation (ICML 2025 · similarity 0.89)
• WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.89)
• DELTA-TTS: Adapting Autoregressive Model into a Diffusion Language Model for Text-to-Speech (NeurIPS 2026 · similarity 0.89)
• DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors (ICLR 2025 · similarity 0.88)
• ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.88)