Research map / Image generation / Fast diffusion sampling
Predictive–Generative Drift Decomposition for Speech Enhancement and Separation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SIPS, a plug-and-play speech enhancement and separation method that combines a pretrained predictor's output as a deterministic drift with a clean-speech-only score/denoiser prior inside a stochastic interpolant SDE sampler.
Problem
Predictive speech enhancement/separation models give high signal-level fidelity but sound unnatural in hard conditions, while generative models produce natural speech but hallucinate and lose faithfulness; existing hybrids are tied to specific predictors or degradations.
Approach
Decompose the stochastic interpolant dynamics into a predictor-induced drift (P(y)-y) and a learned denoiser term trained on clean speech only, so any pretrained predictor can be plugged into SDE sampling with few steps and a degradation-agnostic prior.
Key results
- On WHAMR! with FlexIO, SIPS raises NISQA from 3.54 to 4.01 (+0.47) and UTMOS from 2.79 to 3.01 with SI-SDR 8.51 vs 8.45 dB.
- On VoiceBank-DEMAND with SEMamba, SIPS keeps SI-SDR at 19.63 dB and PESQ 3.43 while improving UTMOS to 4.09 and reducing WER to 8.81% vs 8.87%.
- On EARS-WHAM mismatched condition with SEMamba, SIPS improves NISQA 3.57 to 3.82 and UTMOS 2.93 to 2.97 with SI-SDR 11.28 dB vs 11.36 dB.
Surprising findings
- Diffiner degrades substantially with only 15 sampling steps, while SIPS maintains performance at 15 steps.
- StoRM with a mismatched predictor (NCSN++-trained score on SEMamba outputs) drops SI-SDR sharply, e.g. 19.72 to 12.49 dB.
Limitations
- Intrusive metrics such as SI-SDR and PESQ slightly decrease for some predictors (e.g. PESQ 3.56 to 3.43 for SEMamba on VoiceBank).
- WER increases in the mismatched condition and for some hybrids, attributed to generative hallucinations.
- SepFormer required an oracle scale compensation not applicable in practice.
Experimental setup
Datasets: VoiceBank-DEMAND WHAMR! EARS-WHAM v2
Benchmarks: VoiceBank-DEMAND EARS-WHAM (v2) WHAMR!
Metrics: SI-SDR PESQ DNSMOS P.808 NISQA UTMOS WER
Baselines: SGMSE+ StoRM Diffiner SepFormer FlexIO
Builds on: stochastic interpolants EDM2 StoRM Diffiner DDRM
Closest papers by meaning
- DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion (AAAI 2024 · similarity 0.88)
- Shallow Diffusion for Fast Speech Enhancement (Student Abstract) (AAAI 2024 · similarity 0.87)
- DiffAR: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generation (ICLR 2024 · similarity 0.87)
- FINALLY: fast and universal speech enhancement with studio-like quality (NeurIPS 2024 · similarity 0.86)
- Is Noise Conditioning Necessary for Denoising Generative Models? (ICML 2025 · similarity 0.86)
- Generative Pre-training for Speech with Flow Matching (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.