Research map / Label-efficient and robust learning / Recognition and long-tail
ProsodyFM: Unsupervised Phrasing and Intonation Control for Intelligible Speech Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProsodyFM, a flow-matching TTS model with a Phrase Break Encoder, Duration Predictor, and Terminal Intonation Encoder with Pitch Processor, trained without prosodic labels to improve phrasing and intonation intelligibility.
Problem
Current TTS models miss or misplace phrase breaks in long, complex sentences and produce unnatural terminal intonation, hurting speech intelligibility; break labels are costly and pitch tracking from absolute values is unreliable.
Approach
Model phrase breaks via a break detector-derived initial location plus a learned duration predictor, and model terminal intonation as the shape of smoothed, offset-perturbed pitch contours selected from a learnable bank of intonation shape tokens via attention, all trained end-to-end under an OT-CFM flow-matching backbone without prosodic labels.
Key results
- On LibriTTS test, ProsodyFM achieves RMSE_f0 0.3068, WER 3.22%, F1_break 62.76, outperforming StyleSpeech, GenerSpeech, StyleTTS2 and MatchaTTS on all three objective metrics.
- In the parallel MOS test, ProsodyFM scores 4.22 MOS, 4.47 intelligibility, 4.36 intonation and 4.42 break similarity, versus 4.09 MOS for StyleTTS2 on overall quality.
- In out-of-distribution training on VCTK and testing on LibriTTS, ProsodyFM's WER rises to 4.97% versus 6.78% for MatchaTTS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ProsodyFM instead of StyleTTS2 | TTS model · LibriTTS test | RMSE_f0: 0.3120 -> 0.3068 | Helped |
| Use ProsodyFM instead of StyleTTS2 | TTS model · LibriTTS test | WER: 3.25% -> 3.22% | Helped |
| Use ProsodyFM instead of StyleSpeech | TTS model · LibriTTS test | F1_break: 60.49 -> 62.76 | Helped |
| Use ProsodyFM instead of MatchaTTS | TTS model · LibriTTS OOD test | WER: 6.78% -> 4.97% | Helped |
| Add Terminal Intonation Encoder (ablation) | ProsodyFM · LibriTTS validation | RMSE_f0: 0.3373 -> 0.3047 | Helped |
| Add Terminal Intonation Encoder (ablation) | ProsodyFM · LibriTTS validation | WER: 3.13% -> 2.86% | Helped |
| Add Terminal Intonation Encoder (ablation) | ProsodyFM · LibriTTS validation | F1_break: 61.25% -> 62.51% | Helped |
| Add Phrase Break Encoder (ablation) | ProsodyFM · LibriTTS validation | RMSE_f0: 0.3355 -> 0.3047 | Helped |
| Add Phrase Break Encoder (ablation) | ProsodyFM · LibriTTS validation | WER: 3.10% -> 2.86% | Helped |
| Add Phrase Break Encoder (ablation) | ProsodyFM · LibriTTS validation | F1_break: 61.06% -> 62.51% | Helped |
| Add Terminal Intonation Encoder and Phrase Break Encoder (ablation) | ProsodyFM · LibriTTS validation | RMSE_f0: 0.3391 -> 0.3047 | Helped |
| Add Terminal Intonation Encoder and Phrase Break Encoder (ablation) | ProsodyFM · LibriTTS validation | WER: 3.33% -> 2.86% | Helped |
Surprising findings
- StyleTTS2 achieves higher overall MOS quality than ProsodyFM in the non-parallel setting (4.26 vs 4.07), which the authors attribute to ProsodyFM's smaller training set.
Limitations
- Trained on a relatively small dataset (71 hours of LibriTTS), which likely limits overall speech quality.
- RMSE_f0 is computed only on voiced frames, so ablations show little difference on that metric.
- Subjective labels for appropriateness were replaced by similarity-to-reference judgments, so prosody appropriateness is not directly assessed.
Experimental setup
Datasets: LibriTTS VCTK
Benchmarks: LibriTTS
Metrics: RMSE_f0 WER F1_break MOS MOS_intelligibility MOS_intonation MOS_break
Baselines: StyleSpeech GenerSpeech StyleTTS2 MatchaTTS
Builds on: MatchaTTS OT-CFM Glow-TTS HiFi-GAN PSST
Closest papers by meaning
- EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge (NeurIPS 2025 · similarity 0.87)
- VowelPrompt: Hearing Speech Emotions from Text via Vowel-level Prosodic Augmentation (ICLR 2026 · similarity 0.87)
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.87)
- Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation (NeurIPS 2024 · similarity 0.86)
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark (ICLR 2026 · similarity 0.86)
- Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbing (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.