Research map / Image generation / Fast diffusion sampling

DELTA-TTS: Adapting Autoregressive Model into a Diffusion Language Model for Text-to-Speech

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Masked diffusion language model decoding

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Converts the pretrained AR TTS model CosyVoice3 into a masked discrete diffusion language model via LoRA, a convolution module, a 1/t-weighted loss and a time-shifted unmasking schedule, achieving 1.75% WER on Seed-TTS test-en with 3.3x faster token generation.

Problem

Autoregressive TTS models generate speech tokens sequentially, causing linear inference latency and error propagation that can lead to hallucination; AR-to-dLLM conversion has not been explored for speech.

Approach

Adapt a frozen pretrained AR TTS backbone into a discrete diffusion LM with bidirectional attention, block-wise LoRA plus a Conformer-style depthwise convolution, a 1/t-weighted masked diffusion loss with shift operation, and a time-shifted confidence-ordered unmasking schedule.</newline>

Key results

• DELTA-TTS achieves 1.75% WER on Seed-TTS test-en versus 2.02% for its CosyVoice3 backbone, with 0.144 RTF (3.3x speedup at token-generation stage).
• On LibriSpeech-PC Subset B, DELTA-TTS reaches 2.17% WER, lower than every NAR baseline and second only to CosyVoice3 (1.95%).
• Naive text-domain conversion gives 3.01% WER; adding the convolution module drops it to 1.61%, and a capacity-matched LoRA-only control reaches 2.13%.

Surprising findings

• Naive AR-to-dLLM conversion degrades WER to 3.01%, and the convolution module is needed to recover and surpass the backbone.
• The AR backbone (CosyVoice3) collapses completely (WER>=80%) on 5 of 5440 generations, while DELTA-TTS never does.

Limitations

• Target length is determined by a rule-based audio-to-character rate, a general challenge for dLLM-based generation.
• Operates only in English; multilingual extension is future work.
• Full fine-tuning overfits on the limited 585-hour adaptation data.

Experimental setup

Datasets: LibriTTS Seed-TTS test-en LibriSpeech-PC test-clean Subset B CV3-eval

Benchmarks: Seed-TTS test-en LibriSpeech-PC test-clean Subset B

Metrics: WER SIM UTMOS RTF CMOS SMOS

Baselines: Seed-TTS CosyVoice CosyVoice2 Spark TTS FireRedTTS2 IndexTTS2 VoxCPM Llasa CosyVoice3 MaskGCT E2 TTS F5-TTS

Builds on: CosyVoice3 LoRA masked discrete diffusion (LLaDA-style) Conformer MaskGCT time-shifted schedule

Closest papers by meaning

• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.89)
• DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation (ICML 2025 · similarity 0.89)
• Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.89)
• From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs (NeurIPS 2026 · similarity 0.89)
• DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors (ICLR 2025 · similarity 0.89)
• TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling (NeurIPS 2025 · similarity 0.89)