Research map / Image generation / Autoregressive and latent generation
NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NaturalSpeech 2, a zero-shot TTS system that uses a neural codec with continuous latent vectors (RVQ-approximated) and a non-autoregressive latent diffusion model with speech prompting, trained on 44K hours of speech and ~30h singing.
Problem
Large-scale zero-shot TTS built on discrete codec tokens and autoregressive language models suffers from unstable prosody, word skipping/repeating, and poor voice quality, and the codec/LM dilemma limits both reconstruction and generation.
Approach
Replace discrete tokens with continuous codec latent vectors (summed RVQ embeddings) and generate them with a non-autoregressive latent diffusion model conditioned on a phoneme prior with duration/pitch predictors, using speech prompts (no text prompt needed) for in-context zero-shot learning.
Key results
- NaturalSpeech 2 achieves WER 2.26 on LibriSpeech (ground truth 1.94) and 6.99 on VCTK (ground truth 9.49), versus YourTTS 7.10 and 14.80.
- CMOS vs ground truth is +0.04 on LibriSpeech and -0.30 on VCTK; NaturalSpeech 2 beats YourTTS by 0.65 and 0.58 CMOS.
- On the 50 hard sentences, NaturalSpeech 2 has 0% error rate, while Tacotron and Transformer TTS have 24% and 34%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NaturalSpeech 2 instead of YourTTS | zero-shot TTS · LibriSpeech | CMOS: -0.65 -> 0.00 | Helped |
| Use NaturalSpeech 2 instead of YourTTS | zero-shot TTS · VCTK | CMOS: -0.58 -> 0.00 | Helped |
| Use NaturalSpeech 2 instead of YourTTS | zero-shot TTS · LibriSpeech | SMOS: 2.03 -> 3.28 | Helped |
| Use NaturalSpeech 2 instead of YourTTS | zero-shot TTS · VCTK | SMOS: 2.43 -> 3.20 | Helped |
| Use NaturalSpeech 2 instead of YourTTS | zero-shot TTS · LibriSpeech | WER: 7.10 -> 2.26 | Helped |
| Use NaturalSpeech 2 instead of YourTTS | zero-shot TTS · VCTK | WER: 14.80 -> 6.99 | Helped |
| Use NaturalSpeech 2 instead of YourTTS | zero-shot TTS · LibriSpeech | Pitch mean difference with prompt: 10.52 -> 10.11 | Helped |
| Use NaturalSpeech 2 instead of YourTTS | zero-shot TTS · VCTK | Pitch mean difference with prompt: 13.67 -> 13.29 | Helped |
| Use NaturalSpeech 2 instead of VALL-E | zero-shot TTS · VALL-E demo samples | SMOS: 3.53 -> 3.83 | Helped |
| Use NaturalSpeech 2 instead of VALL-E | zero-shot TTS · VALL-E demo samples | CMOS: -0.31 -> 0.00 | Helped |
| Add duration/pitch prompting (ablation) | NaturalSpeech 2 · LibriSpeech | Pitch mean difference with prompt: 21.69 -> 10.11 | Helped |
| Add diffusion prompt attention (ablation) | NaturalSpeech 2 · LibriSpeech | Pitch mean difference with prompt | Helped |
Surprising findings
- Removing the diffusion speech prompt causes training to fail to converge entirely.
- Applying cross-attention directly to the prompt hidden sequence leaks details and misleads generation, motivating the query-token attention design.
Limitations
- Model is still underfitting at 300K steps according to the authors
- Diffusion inference requires many steps (150 for speech, 1000 for singing), adding latency
- Broader impacts: risk of voice spoofing and impersonation
Experimental setup
Datasets: Multilingual LibriSpeech (English) LibriSpeech test-clean VCTK Singing data crawled from the web (~30 hours)
Benchmarks: LibriSpeech test-clean VCTK 50 hard sentences from FastSpeech
Metrics: CMOS SMOS WER prosody similarity (mean/std/skew/kurt of pitch and duration intelligibility error rate Pearson correlation and RMSE of prosody
Baselines: YourTTS VALL-E Tacotron Transformer TTS FastSpeech NaturalSpeech
Builds on: SoundStream NaturalSpeech WaveNet FiLM Score-based SDE
Closest papers by meaning
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (ICML 2024 · similarity 0.92)
- Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis (ICLR 2024 · similarity 0.90)
- SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation (ICML 2025 · similarity 0.89)
- EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge (NeurIPS 2025 · similarity 0.89)
- DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors (ICLR 2025 · similarity 0.89)
- DELTA-TTS: Adapting Autoregressive Model into a Diffusion Language Model for Text-to-Speech (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.