Research map / Vision-language and multimodal / CLIP and zero-shot
Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Mega-TTS 2 is a zero-shot TTS framework that decouples prosody and timbre into a compressive acoustic autoencoder and uses a prosody language model over multi-sentence prompts plus probability-level prosody interpolation to enable long-prompt cloning and controllable prosody transfer.
Problem
Zero-shot TTS prompting uses single-sentence prompts, so it cannot exploit extra data at inference, and prosody is entangled with timbre, which prevents controllable prosody transfer.
Approach
Decompose speech into content, timbre and prosody; compress prosody into a small VQ codebook modeled by an autoregressive prosody LLM (P-LLM) trained on concatenated multi-sentence prompts; encode timbre with a multi-reference timbre encoder; and perform prosody interpolation by mixing next-token distributions from two P-LLM contexts.
Key results
- Ours-300s reaches 2.23% WER, 0.932 SIM and 4.12 QMOS on LibriSpeech test-clean, versus 3.11% WER, 0.934 SIM and 4.08 QMOS for the fine-tuned baseline with 300s data.
- With 3s prompts, Mega-TTS 2 achieves 2.46% WER and 0.898 SIM versus 5.83% WER and 0.885 SIM for VALL-E.
- In prosody transfer on ESD-to-LibriSpeech, Mega-TTS 2 gets the lowest duration error (32.8) and closest pitch moments to ground truth among CopyCat and Daft-Exprt.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase timbre prompt length from 10s to 60s or 300s (ablation) | Mega-TTS 2 · LibriSpeech test-clean | SIM: 0.905 -> 0.922; 0.905 -> 0.930 | Helped |
| Increase prosody prompt length from 10s to 60s or 300s (ablation) | Mega-TTS 2 · LibriSpeech test-clean | DTW: 32.30 -> 30.74; 32.30 -> 30.25 | Helped |
| Remove MRTE and prosody-timbre disentanglement (ablation) | Mega-TTS 2 · LibriSpeech test-clean | SIM: 0.905 -> 0.841 | Hurt |
| Replace VQ encoder and P-LLM with VAE prosody modeling (ablation) | Mega-TTS 2 · LibriSpeech test-clean | SIM: 0.905 -> 0.896 | Hurt |
| Replace VQ encoder and P-LLM with VAE and latent diffusion modeling (ablation) | Mega-TTS 2 · LibriSpeech test-clean | SIM: 0.905 -> 0.907 | No clear effect |
| Set VQ encoder compression rate to r=2 rather than r=8 (ablation) | Mega-TTS 2 · LibriSpeech test-clean | WER: 2.46% -> 5.53% | Hurt |
| Set VQ encoder compression rate to r=4 rather than r=8 (ablation) | Mega-TTS 2 · LibriSpeech test-clean | WER: 2.46% -> 4.01% | Hurt |
| Set VQ encoder compression rate to r=16 rather than r=8 (ablation) | Mega-TTS 2 · LibriSpeech test-clean | WER: 2.46% -> 2.62% | Hurt |
| Set VQ encoder compression rate to r=32 rather than r=8 (ablation) | Mega-TTS 2 · LibriSpeech test-clean | WER: 2.46% -> 3.57% | Hurt |
| Set MRTE downsampling rate to d=8 rather than d=16 (ablation) | Mega-TTS 2 · LibriSpeech test-clean | SIM: 0.898 -> 0.900 | Helped |
| Set MRTE downsampling rate to d=32 rather than d=16 (ablation) | Mega-TTS 2 · LibriSpeech test-clean | SIM: 0.898 -> 0.880 | Hurt |
| Use phoneme-level or word-level instead of stride-8-level prosody modeling (ablation) | Mega-TTS 2 · LibriSpeech test-clean | WER: 2.46% -> 2.94%; 2.46% -> 3.12% | Hurt |
Surprising findings
- VALL-E performance drops sharply at 20-second prompts (8.77% WER) because it is trained on single-sentence prompts and cannot handle longer prompts.
- Removing the multi-reference timbre encoder raises WER to 5.57% and drops SIM to 0.841, indicating timbre absorbed by the VQ codebook burdens the P-LLM.
Limitations
- Trained only on English data; does not support multilingual TTS.
- Speech quality limited by training data fidelity.
- Attention window design could further improve in-context learning of P-LLM.
Experimental setup
Datasets: LibriLight LibriSpeech test-clean ESD LibriSpeech 960h
Benchmarks: LibriSpeech test-clean ESD LibriSpeech test-other
Metrics: WER SIM DTW QMOS SMOS CMOS DE pitch moments
Baselines: VALL-E Baseline fine-tune (FastSpeech 2 with GAN) CopyCat Daft-Exprt VALL-E-20s
Builds on: VALL-E FastSpeech 2 HiFi-GAN CVQ-VAE
Closest papers by meaning
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.90)
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (ICML 2024 · similarity 0.89)
- Enhancing Zero-Shot Multi-Speaker TTS with Negated Speaker Representations (AAAI 2024 · similarity 0.89)
- CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR 2024 · similarity 0.89)
- StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching (AAAI 2025 · similarity 0.89)
- Do Not Mimic My Voice : Speaker Identity Unlearning for Zero-Shot Text-to-Speech (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.