Research map / Large language models / Language, safety and interpretability
EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds EmergentTTS-Eval, a 1,645-prompt TTS benchmark across six challenging categories generated by iterative LLM refinement, and uses Gemini 2.5 Pro as a pairwise LALM judge that correlates with human preference.
Problem
Existing TTS benchmarks use restricted text domains, lack linguistic diversity, and rely on costly, non-reproducible human evaluations, so they fail to probe nuanced, semantically complex text such as emotion, paralinguistics, code-switching, syntax and complex pronunciation.
Approach
Seed prompts from BASE-TTS are expanded breadth-wise and deepened through three LLM refinement steps per category, and the resulting TTS outputs are compared pairwise by a large audio language model judge (Gemini 2.5 Pro) that returns win-rates against a reference system.
Key results
- Overall win-rates against gpt-4o-mini-tts (Alloy) range from 8.90% to 65.17%, with GPT-4o-audio-preview (Ballad voice, strong prompt) highest.
- Gemini 2.5 Pro judge win-rates show Spearman rank correlation of 90.5% with human judgments.
- Using an LLM (GPT-4.1-mini) for text normalization raised complex-pronunciation win-rate from 51.69% to 76.74%, while WeText normalization did not help.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add category-specific strong prompting (ablation) | GPT-4o-mini-TTS · Overall | win-rate: 56.32% | Helped |
| Add category-specific strong prompting (ablation) | GPT-4o-audio-preview · Overall | win-rate: 53.76% -> 57.95% | Helped |
| Apply WeText text normalization instead of no normalization (ablation) | GPT-4o-mini-TTS · Complex Pronunciation | win-rate: 51.69% -> 50.06% | Hurt |
| Apply GPT-4.1-mini text normalization instead of no normalization (ablation) | GPT-4o-mini-TTS · Complex Pronunciation | win-rate: 51.69% -> 76.74% | Helped |
| Use Qwen 2.5 Omni rather than the other tested LALMs as judge (ablation) | EmergentTTS-Eval judging · TTS system comparisons | win-rate reliability | Hurt |
| Use the tested LALM judges instead of human judges | EmergentTTS-Eval judging · Human preference ranking | Spearman correlation: 90.5% | Helped |
Surprising findings
- Basic text normalization (WeText) did not improve and sometimes harmed TTS win-rate on complex pronunciation.
- Qwen 2.5 Omni as judge produced near-random win-rates (around 50%) with many parsing failures.
Limitations
- LALM judges have inherent biases and can hallucinate pronunciation errors on subjective dimensions.
- Multilingual evaluation uses Latin transcriptions rather than native scripts, and some synthetic refinements are artificial.
Experimental setup
Datasets: BASE-TTS emergent abilities samples (140 seed prompts)
Benchmarks: EmergentTTS-Eval MMAU
Metrics: win-rate WER MOS Spearman correlation Kendall's W
Baselines: gpt-4o-mini-tts (Alloy) Suno Bark Sesame-1B Zonos Tortoise-TTS MiniCPM Qwen 2.5 Omni Orpheus-TTS Deepgram Aura-2 ElevenLabs multilingual v2 HumeAI Octave
Builds on: BASE-TTS emergent abilities test suite EmergentTTS instruction evolution (progressively harder instructions) LLM-as-a-judge
Closest papers by meaning
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.89)
- SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.88)
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark (ICLR 2026 · similarity 0.87)
- Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks (ICLR 2025 · similarity 0.87)
- TTA-Bench: A Comprehensive Benchmark for Evaluating Text-to-Audio Models (AAAI 2026 · similarity 0.87)
- ProsodyFM: Unsupervised Phrasing and Intonation Control for Intelligible Speech Synthesis (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.