Research map / Image generation / Autoregressive and latent generation
DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiTAR, a patch-based autoregressive model that pairs a causal LM over continuous speech patches with a bidirectional diffusion transformer (LocDiT) for intra-patch generation, and applies it to single-stage zero-shot TTS with a new ODE-based temperature sampler and LM guidance.
Problem
Autoregressive generation of continuous speech representations combined with diffusion is either computationally expensive or performs poorly, especially when a causal-attention LM is used with a diffusion head.
Approach
Divide continuous tokens into patches: a causal LM predicts inter-patch context vectors while a bidirectional diffusion transformer (LocDiT) generates each patch, conditioned on historical patches (outpainting-style) and LM output; adds LM guidance by randomly dropping LM output during training, and defines temperature as the time at which noise is injected into a reverse ODE.
Key results
- DiTAR (0.6B, NFE=10) reaches WER 1.78% on LibriSpeech test-clean subset A and 2.39% on subset B, with about 2.75 TFLOPs versus 37-61 TFLOPs for comparable NAR systems.
- On Seed-EN and Seed-ZH, a 1B DiTAR trained on 280k hours reports WER 1.685% and 1.023%, the best among compared systems.
- Subjective CMOS on LibriSpeech subset B is 0.00 for DiTAR versus -0.04 for F5TTS and -0.32 for E2TTS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use historical context patches in LocDiT instead of removing them (ablation) | DiTAR · Seed-EN, patch size 4 | WER (%): 22.874$^\star$ -> \textbf{1.736} | Helped |
| Use historical context patches in LocDiT instead of removing them (ablation) | DiTAR · Seed-EN, patch size 4 | SIM: 0.56$^\star$ -> \textbf{0.72} | Helped |
| Increase language model size ×4 (ablation) | DiTAR (0.4B) · Seed-EN | WER (%): 1.876 -> \textbf{1.695} | Helped |
| Increase LocDiT size ×4 (ablation) | DiTAR (0.4B) · Seed-EN | WER (%): 1.876 -> \textbf{1.785} | Helped |
| Increase encoder size ×4 (ablation) | DiTAR (0.4B) · Seed-EN | WER (%): 1.876 -> 1.821 | No clear effect |
| Use 2 historical patches instead of 1 (ablation) | DiTAR · Seed-EN, patch size 4 | WER (%): 1.736 -> \textbf{1.809} | Mixed |
| Vary patch size away from the chosen size of 4 (ablation) | DiTAR · Seed-EN | WER (%) | Hurt |
| Use LM guidance rather than setting guidance scale w=0 (ablation) | DiTAR · Seed-EN | WER (%) | Helped |
| Increase guidance scale excessively (ablation) | DiTAR · Seed-EN | WER (%) | Hurt |
| Increase temperature τ (ablation) | DiTAR · Seed-EN | SIM: 0.719 -> 0.727 | Helped |
| Increase temperature τ (ablation) | DiTAR · Seed-EN | WER (%): 1.623 -> 1.689 | Hurt |
| Use DiTAR instead of NaturalSpeech 3 | zero-shot TTS · LibriSpeech test-clean A | WER (%): 1.81 -> 1.78 | Helped |
Surprising findings
- Causal-attention AR with diffusion loss performs poorly at very small patch sizes, since LocDiT loses bidirectional context.
- Without historical patch context, the model fails badly (WER 22.9% at patch size 4, many samples fail to stop).
Limitations
- Single-stage TTS evaluated mainly on English and Chinese speech benchmarks
- Latency/RTF trade-off depends on patch size
- Larger-scale results rely on proprietary or internal data
Experimental setup
Datasets: Librilight Emilia
Benchmarks: LibriSpeech test-clean Seed-EN Seed-ZH
Metrics: WER SIM UTMOS FLOPs N-MOS Q-MOS S-MOS CMOS
Baselines: VALL-E MegaTTS 2 NaturalSpeech 3 NaturalSpeech 2 Voicebox MaskGCT E2TTS F5TTS CosyVoice 2 Seed-TTS
Builds on: MAR Transfusion DiT F5-TTS Voicebox
Closest papers by meaning
- DiffAR: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generation (ICLR 2024 · similarity 0.91)
- DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors (ICLR 2025 · similarity 0.89)
- DELTA-TTS: Adapting Autoregressive Model into a Diffusion Language Model for Text-to-Speech (NeurIPS 2026 · similarity 0.89)
- The Design Space of Tri-Modal Masked Diffusion Models (NeurIPS 2026 · similarity 0.89)
- DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation (NeurIPS 2024 · similarity 0.89)
- Energy-Based Diffusion Language Models for Text Generation (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.