Research map / Label-efficient and robust learning / Recognition and long-tail

Rethinking Training Targets, Architectures and Data Quality for Universal Speech Enhancement

NeurIPS 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Multilingual language representation pretraining · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Revisits dereverberation targets for universal speech enhancement, showing time-shifted anechoic speech beats early-reflected targets, and proposes a regression-then-GAN residual two-stage framework plus VQScore-based data filtering, with released weights on Hugging Face.

Problem

Universal speech enhancement under diverse degradations faces unresolved issues in training target choice for dereverberation, the distortion-perception tradeoff (over-smoothing vs hallucination), and the effect of noisy large-scale training data.

Approach

Use time-shifted anechoic clean speech s[n-n0] as the dereverberation target instead of early-reflected speech, and combine a frozen regression model (fidelity) with a residual GAN generator (perceptual correction), motivated by distortion-perception theory and optimal transport; additionally filter training data with VQScore.

Key results

• Shifted anechoic target raises UTMOS from 2.26 to 2.76 and CAcc from 87.62 to 89.41 versus early-reflected target on the URGENT 2025 non-blind test set.
• Adding GAN correction to the shifted anechoic model reaches NISQA 4.12, UTMOS 2.80 and CAcc 89.88, the best non-intrusive and CAcc scores in the table.
• On 44.1 kHz subset, proposed model reaches UTMOS 2.68 and CAcc 92.20 versus Resemble Enhance UTMOS 2.11 and CAcc 47.20.

Surprising findings

• Early-reflected dereverberation targets degrade perceptual quality and ASR accuracy compared with time-shifted anechoic targets.
• Models trained on the organizer-filtered URGENT data still learn to preserve residual background noise and hiss, imposing a performance ceiling.

Limitations

• Non-causal design makes the model unsuitable for real-time applications.
• Long-duration packet loss concealment produces incorrect phonemes, especially in multilingual settings.
• Reference-based leaderboard metrics are biased by the early-reflected reference mismatch, requiring re-evaluation with anechoic references.

Experimental setup

Datasets: URGENT 2025 training data (CommonVoice, DNS5, MLS, LibriTTS, VCTK, WSJ, EARS) FLEURS CML-TTS Emilia-YODAS

Benchmarks: URGENT 2025 non-blind test set FLEURS (Italian, Dutch, Japanese)

Metrics: DNSMOS NISQA UTMOS PESQ ESTOI SDR MCD LSD SBERT LPS

Baselines: TF-GridNet URGENT 2025 Rank 1 (Bobbsun) URGENT 2025 Rank 2 (rc) URGENT 2025 Rank 3 (Xiaobin) ClearerVoice-Studio Resemble Enhance FLEURS-R

Builds on: URGENT 2025 Challenge USEMamba TF-GridNet Wasserstein GAN Miipher-2

Closest papers by meaning

• FINALLY: fast and universal speech enhancement with studio-like quality (NeurIPS 2024 · similarity 0.88)
• Predictive–Generative Drift Decomposition for Speech Enhancement and Separation (NeurIPS 2026 · similarity 0.86)
• Metis: A Foundation Speech Generation Model with Masked Generative Pre-training (NeurIPS 2025 · similarity 0.85)
• Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech (ICLR 2024 · similarity 0.85)
• Audio Large Language Models Can Be Descriptive Speech Quality Evaluators (ICLR 2025 · similarity 0.85)
• Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.85)