Research map / Image generation / Autoregressive and latent generation
Neural text-to-speech synthesis
Synthesize speech from text with codec, diffusion, and vector-quantized generative models.
72 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Autoregressive and latent generation, keyword cluster “speech · music · audio”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
16 papers in 2023–24 and 38 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 3.4% to 4.2%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Deep Voice 2: Multi-Speaker Neural Text-to-Speech (NeurIPS 2017 · 212 citations)
- 2019 · FastSpeech: Fast, Robust and Controllable Text to Speech (NeurIPS 2019 · 579 citations)
- 2020 · Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search (NeurIPS 2020 · 199 citations)
- 2021 · Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis (ICLR 2021 · 81 citations)
- 2022 · Translatotron 2: High-quality direct speech-to-speech translation with voice preservation (ICML 2022 · 20 citations)
- 2023 · QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation (CVPR 2023 · 45 citations)
- 2024 · DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation (CVPR 2024 · 49 citations)
- 2025 · TechSinger: Technique Controllable Multilingual Singing Voice Synthesis via Flow Matching (AAAI 2025 · 10 citations)
- 2026 · IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech (AAAI 2026 · 6 citations)
Earliest papers here
- Deep Voice 2: Multi-Speaker Neural Text-to-Speech (NeurIPS 2017 · 212 citations)
- Dual Supervised Learning (ICML 2017 · 72 citations)
- FastSpeech: Fast, Robust and Controllable Text to Speech (NeurIPS 2019 · 579 citations)
What has been tried
47 papers here report 552 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 8%, hurt 23%, mixed 1%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use expression-to-gesture condition flow rather than naïve concatenation | BEAT test · FMD | 354.60 -> 324.67 | Helped | DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation (CVPR 2024 · 49 citations) |
| Add duration/pitch prompting | LibriSpeech · Pitch mean difference with prompt | 21.69 -> 10.11 | Helped | NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · 37 citations) |
| Replace vq-wav2vec tokens with HuBERT tokens | LibriTTS test set B resynthesis · WER | 3.64 -> 2.73 | Helped | UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (AAAI 2024 · 27 citations) |
| use factorization in the codec and generation rather than removing it | LibriSpeech test-clean · Sim-O | 0.55 -> 0.67 | Helped | NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (ICML 2024 · 20 citations) |
| use motion-removed identity features instead of source image features including motion | LRS2 · C-SIM | 0.451 -> 0.593 | Helped | Faces that Speak: Jointly Synthesising Talking Face and Speech from Text (CVPR 2024 · 11 citations) |
| Add Squeezeformer instead of using convolution in the technique detector | Technique detector · F1 | 0.742 -> 0.770 | Helped | TechSinger: Technique Controllable Multilingual Singing Voice Synthesis via Flow Matching (AAAI 2025 · 10 citations) |
| use LLM-predicted gesture types for semantic retrieval instead of no retrieval | BEAT2 test, all speakers · BeatAlign | 0.447 -> 0.514 | Helped | Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesis (CVPR 2025 · 8 citations) |
| Add the linguistic predictor | Lip2Wav · WER | 34.71 -> 42.51 | Helped | Let There Be Sound: Reconstructing High Quality Speech from Silent Videos (AAAI 2024 · 4 citations) |
| add short monologue training data | Fisher monologue test · WER | 28.50 -> 11.99 | Helped | CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations (NeurIPS 2024 · 3 citations) |
| add DVM to EmoReg | ESD English, average emotion similarity · emotion similarity | 0.94 ± 0.004 -> 0.96 ± 0.003 | Helped | EmoReg: Directional Latent Vector Modeling for Emotional Intensity Regularization in Diffusion-based Voice Conversion (AAAI 2025 · 1 citation) |
| Remove monologue pretraining | Two-speaker test set · WER | 5.67 -> 82.66 | Hurt | CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching (NeurIPS 2025 · 1 citation) |
| Remove curriculum learning | Ablation study · WER (%) | 20.78 -> 27.52 | Hurt | Emotional Face-to-Speech (ICML 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 17 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- LibriSpeech · 5 of 17 papers
- LibriLight · 3 of 17 papers
- LibriTTS · 3 of 17 papers
- Emilia · 3 of 17 papers
- VCTK · 3 of 17 papers
- ESD · 2 of 17 papers
- RAVDESS · 2 of 17 papers
- AudioSet · 2 of 17 papers
Baselines
- VoiceCraft · 4 of 17 papers
- VALL-E · 4 of 17 papers
- Voicebox · 4 of 17 papers
- NaturalSpeech 3 · 3 of 17 papers
- CosyVoice 2 · 3 of 17 papers
- EmoVox · 2 of 17 papers
- StyleSinger · 2 of 17 papers
- CLaM-TTS · 2 of 17 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 54 papers; from OpenAlex and the papers' affiliations).
- The Chinese University of Hong Kong · 6 papers
- Zhejiang University · 5 papers
- Microsoft · 5 papers
- Shanghai Jiao Tong University · 4 papers
- NVIDIA · 3 papers
- University of Science and Technology of China · 3 papers
- Korea Advanced Institute of Science and Technology · 2 papers
- Huawei · 2 papers
- Tsinghua University · 2 papers
- Meta · 2 papers
Use it
Download BibTeX of 35 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Diffusion audio synthesis (52 papers)
- Controllable music generation (42 papers)
Elsewhere in the topic
- Classical generative image synthesis (147 papers)
- High-resolution diffusion synthesis (140 papers)
- Retrieval and matching with generative features (140 papers)
- Generative latent space tooling (129 papers)
- Generalizable fake image detection (119 papers)
- Latent diffusion with learned tokenizers (114 papers)
- Synthetic training data from generators (111 papers)
- Compositional scene generation (105 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).