Research map / Image generation / Autoregressive and latent generation
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NaturalSpeech 3, a zero-shot TTS system that factorizes speech with a neural codec (FACodec) into content, prosody, timbre and acoustic-detail subspaces and generates each with a factorized discrete diffusion model.
Problem
Large-scale zero-shot TTS still lags in speech quality, speaker similarity, and prosody because speech entangles many attributes that are hard to model jointly in a single token stream.
Approach
A factorized neural codec (FACodec) with factorized vector quantization, information bottleneck, supervision, gradient reversal and detail dropout disentangles speech into content, prosody, acoustic detail and timbre; a non-autoregressive discrete diffusion model then generates duration, prosody, content and acoustic details sequentially, each conditioned on its own attribute prompt, while timbre
Key results
- On LibriSpeech test-clean, NaturalSpeech 3 reaches Sim-O 0.67, SMOS 4.01 and WER 1.81, versus 0.64 and 3.69 and 2.03 for Voicebox (the strongest baseline on Sim-O/SMOS); CMOS -0.08 vs ground truth.
- On RAVDESS prosody evaluation, average MCD is 4.28 vs 4.44 for the best baseline (Mega-TTS 2), with MCD-Acc 0.52 vs 0.40.
- Scaling to 200K hours and 1B parameters improves WER from 2.11 to 1.71 and Sim-O from 0.73 to 0.78 on an internal test set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use factorization in the codec and generation rather than removing it (ablation) | NaturalSpeech 3 · LibriSpeech test-clean | Sim-O: 0.55 -> 0.67 | Helped |
| use classifier-free guidance at inference (ablation) | NaturalSpeech 3 · LibriSpeech test-clean | SMOS: 3.80 -> 4.01 | Helped |
| use the learned codec prosody representation instead of Mel 20 Bins (ablation) | NaturalSpeech 3 · RAVDESS | Average MCD: 4.34 vs 4.28 | Helped |
| reduce each diffusion process from 4 iterations to 1 (ablation) | NaturalSpeech 3 · LibriSpeech test-clean | Sim-O: 0.67 -> 0.66 | Hurt |
| use multi-step rather than one-step duration generation (ablation) | NaturalSpeech 3 duration model · LibriSpeech test-clean | Sim-O: 0.62 -> 0.67 | Helped |
| use classification cross-entropy rather than regression L2 loss (ablation) | NaturalSpeech 3 duration model · LibriSpeech test-clean | WER: 2.38 -> 1.94 | Helped |
| add phoneme-level prosody conditioning (ablation) | NaturalSpeech 3 duration model · LibriSpeech test-clean | WER: 2.49 -> 1.94 | Helped |
| add duration prompting (ablation) | NaturalSpeech 3 duration model · LibriSpeech test-clean | WER: 2.83 -> 1.94 | Helped |
| add the information bottleneck (ablation) | FACodec · Zero-shot voice conversion | Sim-O: 0.73 -> 0.86 | Helped |
| add acoustic details quantizers (ablation) | FACodec · Speech codec reconstruction | PESQ: 3.09 -> 3.47 | Helped |
| scale training data from 1K to 60K hours | NaturalSpeech 3, 500M · Internal test set | Sim-O: 0.64 -> 0.72 | Helped |
| scale training data from 60K to 200K hours | NaturalSpeech 3, 500M · Internal test set | WER: 3.03 -> 2.11 | Helped |
Surprising findings
- Removing classifier-free guidance leaves WER unchanged while lowering CMOS and SMOS, and the authors note Voicebox's SMOS is lower than its Sim-O suggests, indicating a gap between objective similarity and subjective judgments.
- Without acoustic-detail quantizers, FACodec with only three codebooks is comparable to or better than SoundStream with three codebooks, suggesting content, prosody and timbre carry most reconstruction information.
Limitations
- Attributes do not cover all speech aspects, e.g. background sounds and energy are not modeled.
- Trained only on English LibriVox audiobooks, so it lacks diversity and multilingual support.
- FACodec's content disentanglement requires phoneme transcription supervision, limiting scalability.
Experimental setup
Datasets: Librilight LibriSpeech RAVDESS VCTK internal 200K-hour speech dataset
Benchmarks: LibriSpeech test-clean RAVDESS VCTK
Metrics: Sim-O Sim-R WER UTMOS CMOS SMOS MCD MCD-Acc PESQ STOI
Baselines: VALL-E NaturalSpeech 2 Voicebox Mega-TTS 2 UniAudio StyleTTS 2 HierSpeech++ EnCodec HiFi-Codec DAC SoundStream
Builds on: NaturalSpeech 2 SoundStream VALL-E MaskGIT-style masked discrete diffusion classifier-free guidance
Closest papers by meaning
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.92)
- Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis (ICLR 2024 · similarity 0.89)
- CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR 2024 · similarity 0.88)
- DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors (ICLR 2025 · similarity 0.88)
- DiffAR: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generation (ICLR 2024 · similarity 0.88)
- Enhancing Zero-Shot Multi-Speaker TTS with Negated Speaker Representations (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.