Research map / Image generation / Autoregressive and latent generation

DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors

ICLR 2025 · Image generation › Autoregressive and latent generation · approach: Neural text-to-speech synthesis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds DiTTo-TTS, a DiT-based latent diffusion TTS model that avoids phonemes and durations, using a speech length predictor and semantically aligned Mel-VAE latents, scaled to 82K hours and 790M parameters.

Problem

Latent diffusion TTS usually depends on phonemes and durations, which limits data scaling; prior phoneme-free LDM TTS models lag in performance.

Approach

Adapt a Diffusion Transformer with minimal changes (global AdaLN, long skip connection, cross-attention to a pretrained text encoder), predict total speech length instead of per-token durations, and align Mel-VAE speech latents with a pretrained LM's embedding space for better text-speech alignment.

Key results

• DiTTo-en-XL reaches WER 1.78 and SIM-o 0.5773 on the LibriSpeech continuation task, versus 2.0 for Voicebox and 2.36 for CLaM-TTS as reported in their papers.
• DiTTo-en-B outperforms CLaM-TTS in WER, CER, and speaker similarity while running 4.6x faster with 3.84x fewer parameters.
• Variable-length speech length predictor modeling reduces cross-sentence WER from 6.81 (fixed-length) to 5.58 (SLP-CE).

Measured effects reported in the paper

Surprising findings

• Sharing a single global AdaLN across layers reduced parameters by 30.5% for XL models and surprisingly improved performance.
• Fine-tuning only the neural audio codec with an auxiliary LM loss (Mel-VAE++) gave a large gain without fine-tuning the text encoder.

Limitations

• Code, weights, and full training implementation are only planned for gradual release subject to legal concerns.
• Many baselines (VALL-E, Voicebox, NaturalSpeech) are not publicly available, so their scores are copied from papers or demo samples, making comparisons less controlled.
• Subjective evaluations use demo samples at differing sampling rates for some baselines.

Experimental setup

Datasets: Multilingual LibriSpeech GigaSpeech LibriTTS-R VCTK LJSpeech Expresso AIHub datasets KsponSpeech LibriSpeech RAVDESS

Benchmarks: LibriSpeech test-clean RAVDESS Multilingual LibriSpeech test subsets

Metrics: WER CER SIM-o SIM-r SMOS CMOS inference time

Baselines: VALL-E SPEAR-TTS CLaM-TTS YourTTS Voicebox Simple-TTS NaturalSpeech 2 NaturalSpeech 3 Mega-TTS E3 TTS StyleTTS 2 XTTS-v2

Builds on: DiT latent diffusion models Mel-VAE Simple-TTS classifier-free guidance

Closest papers by meaning

• DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation (ICML 2025 · similarity 0.89)
• DELTA-TTS: Adapting Autoregressive Model into a Diffusion Language Model for Text-to-Speech (NeurIPS 2026 · similarity 0.89)
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.89)
• TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling (NeurIPS 2025 · similarity 0.88)
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (ICML 2024 · similarity 0.88)
• DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis (ICML 2025 · similarity 0.88)