Research map / Image generation / Fast diffusion sampling
Shallow Flow Matching for Coarse-to-Fine Text-to-Speech Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Shallow Flow Matching (SFM), which builds intermediate states on CondOT flow paths from a weak TTS generator's coarse mel-spectrograms via orthogonal projection and starts flow inference from them, improving naturalness and speeding up adaptive-step ODE solvers.
Problem
Flow-matching TTS coarse-to-fine models start generation from pure noise even though coarse representations from a weak generator already encode much of the speech structure, wasting modeling capacity on early stages.
Approach
Use a lightweight SFM head to predict a time and variance for the coarse mel-spectrogram, orthogonally project it onto CondOT paths to build an intermediate state, and run a single-segment piecewise flow from that state instead of noise, with an SFM strength alpha to scale guidance at inference.
Key results
- On VCTK, Matcha-TTS SFM (alpha=3.5) improves UTMOS from 4.008 (baseline) to 4.106 and CMOS reaches 0.00 vs -0.31 for baseline (significant).
- On LibriTTS, CosyVoice SFM (alpha=2.0) raises UTMOSv2 from 3.303 to 3.480 and CMOS from -0.21 to 0.00 relative to SFM reference.
- With adaptive-step solvers, Matcha-TTS SFM (alpha=4.0) reduces RTF with Heun(2) by 48.5% relative to the ablated model (0.207 vs 0.401).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SFM instead of the coarse-condition ablated method (ablation) | Matcha-TTS · LJ Speech test | UTMOSv2: 3.763 -> 3.848 | Helped |
| Apply SFM instead of the coarse-condition ablated method (ablation) | Matcha-TTS · VCTK test | WER: 1.613 -> 0.952 | Helped |
| Apply SFM instead of the ablated method (ablation) | StableTTS · VCTK test | UTMOSv2: 2.958 -> 3.020 | Helped |
| Add coarse mel-spectrograms as FM conditions (ablation) | StableTTS SFM · VCTK test | UTMOSv2: 3.020 -> 2.899 | Hurt |
| Apply SFM instead of the ablated method (ablation) | CosyVoice · LibriTTS test-clean | UTMOSv2: 3.369 -> 3.480 | Helped |
| Use only speech tokens as the flow encoder input (ablation) | CosyVoice SFM · LibriTTS test-clean | SMOS: 3.67 -> 2.66 | Hurt |
| Add coarse mel-spectrograms as FM conditions (ablation) | CosyVoice-DiT SFM · LibriTTS test-clean | UTMOSv2: 3.171 -> 3.156 | No clear effect |
| Increase SFM strength from α=1.0 to α=2.5 (ablation) | Matcha-TTS SFM · LJ Speech validation | PMOS average: 4.036 -> 4.176 | Helped |
| Increase SFM strength from α=1.0 to α=3.5 (ablation) | Matcha-TTS SFM · VCTK validation | PMOS average: 3.462 -> 3.679 | Helped |
| Increase SFM strength from α=1.0 to α=3.0 (ablation) | StableTTS SFM · VCTK validation | PMOS average: 3.281 -> 3.486 | Helped |
| Increase SFM strength from α=1.0 to α=2.0 (ablation) | CosyVoice SFM · LibriTTS validation | PMOS average: 3.721 -> 4.087 | Helped |
| Increase SFM strength (ablation) | Matcha-TTS SFM with adaptive-step ODE solvers · LJ Speech validation | RTF: 0.401 -> 0.157 | Helped |
Surprising findings
- Using coarse mel-spectrograms as flow conditions (SFM-c) degrades subjective quality or invalidates the method, and the adaptive t_h collapses to 0 in U-Net models.
- The adaptively learned t_h tends to be small, so an SFM strength alpha greater than 1 is needed to improve results.
Limitations
- Uses a simple SFM head, so may be insufficient when the weak generator is unstable.
- Naive concatenation of speech tokens, speaker embeddings and masked mel for CosyVoice encoder input may hurt cross-modal alignment.
- WER and SIM improvements are inconsistent across models.
Experimental setup
Datasets: LJ Speech VCTK LibriTTS
Benchmarks: LJ Speech VCTK LibriTTS
Metrics: UTMOS UTMOSv2 Distill-MOS PMOS WER SIM CMOS SMOS RTF NFE
Baselines: Matcha-TTS CosyVoice StableTTS CosyVoice-DiT Ablated model SFM-c SFM-t
Builds on: Matcha-TTS CosyVoice StableTTS DiffSinger shallow diffusion PeRFlow
Closest papers by meaning
- DELTA-TTS: Adapting Autoregressive Model into a Diffusion Language Model for Text-to-Speech (NeurIPS 2026 · similarity 0.85)
- Kinetic-Optimal Scheduling with Moment Correction for Metric-Induced Discrete Flow Matching in Zero-Shot Text-to-Speech (NeurIPS 2026 · similarity 0.85)
- TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling (NeurIPS 2025 · similarity 0.85)
- Trajectory as the Teacher: Few-Step Discrete Flow Matching via Energy-Navigated Distillation (NeurIPS 2026 · similarity 0.85)
- DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors (ICLR 2025 · similarity 0.84)
- DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.