Research map / Label-efficient and robust learning / Self-supervised and contrastive
Speech Self-Supervised Learning Using Diffusion Model Synthetic Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIFFS4L, which trains a diffusion-based speech synthesizer on a small unannotated speech corpus, generates synthetic speech with new prosody, speakers, and content, and pretrains SSL models (wav2vec2/HuBERT/WavLM) on real plus synthetic speech.
Problem
Self-supervised speech pretraining requires large unannotated corpora, which are impractical for low-resource languages or privacy-constrained settings; existing augmentation mostly perturbs real audio and does not add new prosody, speaker, or content variation.
Approach
Train a DDPM unit-to-speech synthesizer on discrete k-means units from wav2vec2 features plus speaker identity, then generate synthetic utterances at four novelty levels (original, same speaker, novel speaker, novel content via 80% span masking) and pretrain SSL models on the mixture.
Key results
- On English ASR with 100h real speech, HUBERT-DIFFS4L reduces WER by 6.26 points (23.71 to 17.45) relative to HUBERT-REAL.
- DIFFS4L reduces CER by an average of 2.6 and WER by 8.3 points across 13 non-English MLS/CommonVoice languages with WAV2VEC2.
- WAV2VEC2-NOREAL (fully synthetic, no real data) reaches 52.79 WER, showing real data is essential; 100+430+430 gives WER 16.67.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain with DIFFS4L synthetic data instead of real speech alone | WAV2VEC2.0 · English ASR, 100-hour real speech | WER: 23.48 -> 16.67 | Helped |
| Pretrain with DIFFS4L synthetic data instead of real speech alone | HuBERT · English ASR, 100-hour real speech | WER: 23.71 -> 17.45 | Helped |
| Pretrain with DIFFS4L synthetic data instead of real speech alone | WAV2VEC2.0 · English ASR, 960-hour real speech | WER: 10.49 -> 9.93 | Helped |
| Pretrain with DIFFS4L synthetic data instead of real speech alone | HuBERT · English ASR, 960-hour real speech | WER: 10.30 -> 9.87 | Helped |
| Add novel speaker combinations to synthetic speech (ablation) | DIFFS4L / WAV2VEC2.0 · English ASR, 100-hour real speech | CER: 6.91 -> 5.19 | Helped |
| Include real speech in the synthetic-data training mixture (ablation) | DIFFS4L / WAV2VEC2.0 · English ASR, 100-hour real speech | CER: 5.19 -> 18.26 | Helped |
| Use a mixture of SS/NS and NC synthetic speech rather than only one type (ablation) | DIFFS4L / WAV2VEC2.0 · English ASR, 100-hour real speech | CER: 5.58 -> 5.19; 7.88 -> 5.19 | Helped |
| Increase SS/NS synthetic data while holding real data at 100 hours and NC at 0 hours (ablation) | DIFFS4L / WAV2VEC2.0 · English ASR | CER | Mixed |
| Replace diffusion-generated speech with WaveNet-generated speech (ablation) | DIFFS4L / WAV2VEC2.0 · English ASR, 100-hour real speech; 100+430+430 composition | WER: 16.67 -> 21.90 | Hurt |
| Use NC babbles generated by WaveNet instead of diffusion (ablation) | DIFFS4L / WAV2VEC2.0 · English ASR, 100-hour real speech; 100+0+860 composition | WER: 24.91 -> 39.02 | Hurt |
| Combine DIFFS4L with WAVLM | WAVLM · English ASR, 100-hour real speech | WER: 18.38 -> 13.69 | Helped |
| Use the DIFFS4L-augmented data instead of further pretraining on 100 hours of real speech | XLSR-128 · CommonVoice low-resource languages | WER | Helped |
Surprising findings
- Nonsensical babble generated by the diffusion model (novel content) contributes to performance gains, while WaveNet-generated babble only hurts performance.
- Performance vs. synthetic hours is non-monotonic, peaking around 630 hours of SS/NS data and degrading with more synthetic data.
Limitations
- The synthetic data generation pipeline is time-consuming because it requires training multiple networks sequentially.
- DIFFS4L-pretrained models still underperform models pretrained on the same amount of real speech.
- Speaker conversion is limited to speakers seen in the original dataset; no new speakers are introduced.
Experimental setup
Datasets: LibriSpeech-960 Multilingual LibriSpeech CommonVoice
Benchmarks: LibriSpeech 10h ASR SUPERB MLS CommonVoice
Metrics: WER CER accuracy MTWV F1 EER DER
Baselines: WAV2VEC2-REAL HUBERT-REAL WAV2VEC2-AUG WAVLM WaveNet-generated data XLSR-128
Builds on: DDPM wav2vec 2.0 HuBERT ProDiff HiFi-GAN
Closest papers by meaning
- DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation (NeurIPS 2024 · similarity 0.89)
- Scaling Speech-Text Pre-training with Synthetic Interleaved Data (ICLR 2025 · similarity 0.88)
- DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation (ICML 2025 · similarity 0.86)
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.86)
- DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors (ICLR 2025 · similarity 0.86)
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.