Research map / Image generation / Autoregressive and latent generation
Generative Pre-training for Speech with Flow Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pre-trains a 330M-parameter flow-matching Transformer (SpeechFlow) with masked-audio conditioning on 60k hours of unlabeled speech, then fine-tunes it for speech enhancement, separation, and zero-shot TTS.
Problem
Generative models in speech are usually built for a single task via task-specific conditioning or distribution mapping; there is no general-purpose generative pre-trained speech model that can be adapted to multiple downstream generation tasks.
Approach
Pre-train a flow-matching generative model on unlabeled Mel spectrograms with random masked-audio conditioning (dropping to unconditional with some probability), so the pre-trained model can later be fine-tuned by swapping the mask for task-specific conditions such as noisy speech, mixtures, or phone sequences.
Key results
- Fine-tuned SpeechFlow on VoiceBank-Demand reaches PESQ 3.13 and COVL 3.80, matching or exceeding SGMSE+ (2.93 PESQ) and MetricGAN+; on out-of-domain WSJ0-CHiME3 it reaches PESQ 2.70 vs 2.48 for SGMSE+.
- On zero-shot TTS with 960 h labeled data and LoRA, SpeechFlow reaches SIM-o 0.700 and WER 2.1 (32 GPUs), versus Voicebox WER 1.9 and SIM-o 0.662 trained on 60k hours.
- On 2Mix separation (LibriMix), SpeechFlow achieves SI-SDRi 15.85 and ESTOIi 0.37 with learnable inverse-Mel and phase estimation, above Conv-TasNet's 15.24 SI-SDRi and 0.22 ESTOIi.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add generative pre-training (ablation) | SpeechFlow · Voicebank-Demand and WSJ0-CHiME3 speech enhancement | PESQ: 2.92 -> 3.13; 2.38 -> 2.70 | Helped |
| add generative pre-training (ablation) | SpeechFlow · LibriMix separation | SI-SDRi: 11.24 -> 11.74; 10.00 -> 10.46; 8.65 -> 11.08; 7.39 -> 8.22 | Helped |
| add generative pre-training (ablation) | SpeechFlow · LS test-clean zero-shot speaker adaptation TTS | SIM-o: 0.526 -> 0.678 | Helped |
| add generative pre-training (ablation) | SpeechFlow · LS test-clean zero-shot speaker adaptation TTS | WER: 2.3 -> 2.2 | No clear effect |
| use LoRA instead of full-model fine-tuning (ablation) | SpeechFlow, LS test-clean zero-shot speaker adaptation TTS · 32-GPU fine-tuning | SIM-o: 0.697 -> 0.700 | Helped |
| use LoRA instead of full-model fine-tuning (ablation) | SpeechFlow, LS test-clean zero-shot speaker adaptation TTS · 32-GPU fine-tuning | WER: 2.2 -> 2.1 | Helped |
| increase fine-tuning from 1 GPU to 32 GPUs (ablation) | SpeechFlow LoRA · LS test-clean zero-shot speaker adaptation TTS | SIM-o: 0.696 -> 0.700 | Helped |
| increase fine-tuning from 1 GPU to 32 GPUs (ablation) | SpeechFlow LoRA · LS test-clean zero-shot speaker adaptation TTS | WER: 2.6 -> 2.1 | Helped |
| fine-tune jointly on TTS, enhancement, and separation (ablation) | SpeechFlow · Multi-task enhancement | PESQ: 2.87 -> 2.87 | No clear effect |
| fine-tune jointly on TTS, enhancement, and separation (ablation) | SpeechFlow · Multi-task zero-shot speaker adaptation TTS | WER: 2.2 -> 2.3 | Hurt |
| fine-tune jointly on TTS, enhancement, and separation (ablation) | SpeechFlow · Multi-task separation 2 Mix + Noise | SI-SDRi: 12.41 -> 9.73 | Hurt |
| use unconditioned pre-training instead of masked conditioning (ablation) | SpeechFlow · Zero-shot speaker adaptation TTS and enhancement | performance | Hurt |
Surprising findings
- A clean-spectrogram upper bound for separation (SI-SDRi ~12.4 on 2 Mix) was below the Conv-TasNet baseline, showing Mel-spectrogram representation limits SI-SDRi.
- Pre-training improved speaker similarity for TTS but not content accuracy (WER), and unconditioned pre-training (p_cond=0) performed badly.
Limitations
- Mel-spectrogram representation without phase limits waveform quality and metrics such as SI-SDRi.
- Fine-tuning requires frame-aligned conditions; unaligned conditions (raw text, speaker ID) left for future work.
- Only English data was used; multi-task fine-tuning hurt separation performance significantly.
Experimental setup
Datasets: 60k hours English audiobook speech (Libri-Light-style unlabeled) VoiceBank-Demand WSJ0-CHiME3 DNS2020 LibriMix LibriSpeech 960h WHAM! LibriLight
Benchmarks: VoiceBank-Demand WSJ0-CHiME3 LibriMix LibriSpeech test-clean (filtered)
Metrics: PESQ PESQ-nb ESTOI CSIG CBAK COVL SI-SDRi ESTOIi WER SIM-o
Baselines: Conv-TasNet MetricGAN+ SGMSE+ DEMUCS SepFormer YourTTS VALL-E Voicebox A3T
Builds on: Flow Matching Voicebox SGMSE+ HiFi-GAN
Closest papers by meaning
- UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation (ICLR 2025 · similarity 0.88)
- DFlow: A Generative Model Combining Denoising AutoEncoder and Normalizing Flow for High Fidelity Waveform Generation (ICML 2024 · similarity 0.86)
- Predictive–Generative Drift Decomposition for Speech Enhancement and Separation (NeurIPS 2026 · similarity 0.86)
- Metis: A Foundation Speech Generation Model with Masked Generative Pre-training (NeurIPS 2025 · similarity 0.86)
- Scaling Speech-Text Pre-training with Synthetic Interleaved Data (ICLR 2025 · similarity 0.86)
- DiffAR: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generation (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.