Research map / Image generation / Text-to-image diffusion
PromptTTS 2: Describing and Generating Voices with Text Prompt
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PromptTTS 2, a diffusion-based variation network that predicts reference-speech voice representations from text prompts, plus an LLM-based pipeline that auto-generates text prompts from SLU-recognized voice attributes.
Problem
Text-prompt-based TTS suffers from a one-to-many mapping (text prompts cannot capture all voice variability) and a scarcity of human-written speech-prompt datasets.
Approach
A diffusion variation network samples the missing voice-variability representation (extracted from reference speech during training) conditioned on the text prompt representation, so voices can be varied at inference from Gaussian noise; an LLM (GPT-3.5-Turbo) composes text prompts from SLU-recognized attribute labels via placeholder-based sentence templates and combinations.
Key results
- Attribute control mean accuracy 93.33% for PromptTTS 2 vs 91.54% (PromptTTS) and 91.47% (InstructTTS).
- MOS 3.88 ± 0.08 for PromptTTS 2 vs 3.77 (PromptTTS) and 3.80 (InstructTTS); ground truth 4.38.
- Classifier trained on pipeline-generated prompts reaches 96.38% mean accuracy vs 92.74% (human PromptSpeech) and 94.71% (TextrolSpeech).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PromptTTS 2 instead of PromptTTS | Text-prompt TTS on PromptSpeech test · PromptSpeech test | Mean accuracy (%): 91.54 -> 93.33 | Helped |
| Use PromptTTS 2 instead of InstructTTS | Text-prompt TTS on PromptSpeech test · PromptSpeech test | Mean accuracy (%): 91.47 -> 93.33 | Helped |
| Use PromptTTS 2 instead of PromptTTS | Speech synthesis · Speech quality test | MOS: 3.77 ± 0.09 -> 3.88 ± 0.08 | Helped |
| Use PromptTTS 2 instead of InstructTTS | Speech synthesis · Speech quality test | MOS: 3.80 ± 0.07 -> 3.88 ± 0.08 | Helped |
| Train the intention classifier on prompts from the proposed pipeline instead of TextrolSpeech | Prompt generation quality evaluation · PromptSpeech test | Mean intention classification accuracy (%): 94.71 -> 96.38 | Helped |
| Use the proposed pipeline instead of human-authored PromptSpeech prompts | Prompt generation quality evaluation · PromptSpeech test | Mean intention classification accuracy (%): 92.74 -> 96.38 | Helped |
| Use PromptTTS 2 rather than SP-FaceVC for voice generation from facial images | Face2Voice generation · HDTF test speakers | Same-style MOS (%): 20.17 -> 31.78 | Helped |
| Add placeholders to prompt generation (ablation) | Our Prompt Generation Pipeline · PromptSpeech test | Mean intention classification accuracy (%): 94.04 -> 96.38 | Helped |
| Include phrases in prompt generation (ablation) | Our Prompt Generation Pipeline · PromptSpeech test | Mean intention classification accuracy (%): 96.09 -> 96.38 | Helped |
| Include sentences in prompt generation (ablation) | Our Prompt Generation Pipeline · PromptSpeech test | Mean intention classification accuracy (%): 96.16 -> 96.38 | Helped |
Surprising findings
- Baseline systems also show voice variability from text content and TTS backbone sampling, not only from the text prompt.
- Pipeline-generated prompts train a better attribute classifier than human-authored prompts.
Limitations
- Only four attributes (gender, pitch, speed, volume) are covered in the pipeline and experiments.
- Variation network sampling yields limited perceptual variation in human tests (7.41% similarity aspect value in appendix is low but reported as-is) and the face-to-voice extension is only preliminary.
Experimental setup
Datasets: Multilingual LibriSpeech (English) PromptSpeech TextrolSpeech HDTF
Benchmarks: PromptSpeech test set
Metrics: attribute control accuracy MOS CMOS speech similarity (WavLM-TDNN and human) intention classification accuracy
Baselines: PromptTTS InstructTTS SP-FaceVC
Builds on: NaturalSpeech 2 PromptTTS InstructTTS WavLM-TDNN
Closest papers by meaning
- Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis (ICLR 2024 · similarity 0.85)
- The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generation (CVPR 2025 · similarity 0.85)
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.85)
- EPT: Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion (AAAI 2025 · similarity 0.84)
- Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs (ICLR 2026 · similarity 0.84)
- EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.