Research map / Image generation / GANs
FINALLY: fast and universal speech enhancement with studio-like quality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Revisits GANs for universal speech enhancement, arguing LS-GAN training seeks the mode of the clean-speech distribution, and builds FINALLY, a WavLM-perceptual-loss HiFi++ variant with MS-STFT discriminators that outputs 48 kHz speech in one forward pass.
Problem
Universal speech enhancement must restore speech from many simultaneous distortions (noise, reverberation, bandwidth limits, microphone artefacts) at high quality and fast inference, while diffusion-based methods are computationally expensive due to iterative sampling.
Approach
Theoretically shows that LS-GAN training with a deterministic generator drives the output toward the mode of the conditional clean-speech density, and proposes a WavLM convolutional-feature perceptual loss (LMOS, combined with L1 STFT loss) plus MS-STFT discriminators, with a WavLM encoder fed into HiFi++ and a WaveUNet upsampler to produce 48 kHz output.
Key results
- On VoxCeleb real data, FINALLY reaches MOS 4.63 and UTMOS 4.05 versus 4.47 and 3.67 for HiFi-GAN-2, with RTF 0.03 versus 0.50.
- On the UNIVERSE validation set, FINALLY (48 kHz) achieves MOS 4.23 against UNIVERSE's 4.10, with lower PhER (0.14 vs 0.20) and RTF 0.03 vs 0.5.
- On VCTK-DEMAND, FINALLY trails DB-AIAT on PESQ (2.94 vs 3.26) and SI-SDR (4.6 vs 19.3) while leading on MOS, UTMOS and WV-MOS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LMOS regression loss rather than no regression loss (ablation) | FINALLY, stage 2 (small model) · VoxCeleb (real data) | MOS: 3.71 \pm 0.08 -> 4.20 \pm 0.05 | Helped |
| Add WavLM encoder features as input (ablation) | FINALLY · UNIVERSE validation | MOS: 3.49 ± 0.08 -> 3.75 ± 0.07 | Helped |
| Add WavLM encoder features as input (ablation) | FINALLY · UNIVERSE validation | PhER: 0.27 ± 0.04 -> 0.21 ± 0.04 | Helped |
| Scale model by increasing the number of channels (ablation) | FINALLY · VoxCeleb (real data) | MOS: 4.21 ± 0.06 -> 4.30 ± 0.05 | Helped |
| Add Upsample WaveUNet and the 48 kHz third training stage (ablation) | FINALLY-16 · VoxCeleb (real data) | MOS: 4.30 ± 0.05 -> 4.59 ± 0.05 | Helped |
| Add human feedback loss during the third stage (ablation) | FINALLY · VoxCeleb (real data) | UTMOS: 3.78 ± 0.07 -> 4.05 ± 0.07 | Helped |
| Use LMOS rather than RecLoss for regression (ablation) | Small speech enhancement model · VoxCeleb (real data) | MOS: 4.07 \pm 0.06 -> 4.20 \pm 0.05 | Helped |
| Use LMOS rather than MS-STFT regression loss (ablation) | Small speech enhancement model · Regression-loss ablation | UTMOS: 2.54 ± 0.10 -> 3.43 ± 0.09 | Helped |
| Use LMOS rather than RecLoss (ablation) | Small speech enhancement model · Regression-loss ablation | UTMOS: 2.53 ± 0.10 -> 3.43 ± 0.09 | Helped |
| Use LMOS rather than L1Spec regression loss (ablation) | Small speech enhancement model · Regression-loss ablation | UTMOS | Helped |
| Combine WavLM-conv feature loss with L1-STFT loss (ablation) | HiFi-GAN vocoder · Neural vocoding | MOS: 4.31 ± 0.08 | Helped |
Surprising findings
- On VCTK-DEMAND, the model scores far lower on reference-based metrics (PESQ 2.94, SI-SDR 4.6 dB) than baselines, despite higher MOS and UTMOS.
- In vocoding, WavLM convolutional features outperform transformer-layer features, and a waveform-based feature space failed to train at all.
Limitations
- Perceptual quality degrades for severely degraded inputs, with residual artefacts.
- Not suited to low-latency streaming because WavLM's context window sets a minimum latency.
- Large parameter count (454M including 358M WavLM) compared with baselines, and weaker reference-based metrics (PESQ, STOI, SI-SDR).
Experimental setup
Datasets: LibriTTS-R DAPS-clean DNS Challenge noises VoxCeleb1 UNIVERSE validation data VCTK-DEMAND LJSpeech Miipher test samples
Benchmarks: VoxCeleb1 real samples UNIVERSE validation set VCTK-DEMAND LibriTTS test-other (Miipher comparison)
Metrics: MOS UTMOS WV-MOS DNSMOS PhER WER PESQ STOI SI-SDR RTF
Baselines: UNIVERSE BBED STORM VoiceFixer DEMUCS HiFi-GAN-2 MetricGAN+ DB-AIAT Miipher
Builds on: HiFi++ HiFi-GAN MS-STFT discriminator WavLM LS-GAN
Closest papers by meaning
- High Fidelity Speech Synthesis with Adversarial Networks (ICLR 2020 · similarity 0.88)
- Rethinking Training Targets, Architectures and Data Quality for Universal Speech Enhancement (NeurIPS 2026 · similarity 0.88)
- Predictive–Generative Drift Decomposition for Speech Enhancement and Separation (NeurIPS 2026 · similarity 0.86)
- GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling (ICLR 2025 · similarity 0.85)
- Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech (ICLR 2024 · similarity 0.84)
- RealMAN: A Real-Recorded and Annotated Microphone Array Dataset for Dynamic Speech Enhancement and Localization (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.