Research map / Label-efficient and robust learning / Self-supervised and contrastive
Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VQScore, a self-supervised speech quality metric from VQ-VAE quantization error trained only on clean LibriSpeech, plus a VQ-VAE-based self-supervised speech enhancement model using self-distillation and adversarial training.
Problem
Speech quality estimation and speech enhancement typically require labeled quality scores or noisy-clean pairs; existing supervised estimators generalize poorly and labels are costly.
Approach
Use the quantization error (cosine similarity between encoder outputs and nearest codebook entries) of a VQ-VAE trained only on clean speech as a quality score, and reuse the VQ codebook for self-supervised enhancement, made robust via self-distillation with adversarial training of the encoder.
Key results
- VQScore (cos, z) reaches LCC 0.8386 with DNSMOS OVRL on VoiceBank-DEMAND noisy test set, and 0.8604 with real MOS on IUB_voices, versus 0.8181 for DNSMOS P.835 on that subset.
- On IUB_cosine, VQScore (cos, z) achieves 0.4880 LCC with real scores, above DNSMOS (0.3938) and SpeechLMScore (0.3913).
- Proposed+AT enhancement on DNS1 Reverb achieves OVRL 2.456 versus 2.409 for supervised CNN-Transformer and 2.175 for Demucs, trained only on clean speech.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| calculate the VQScore with cosine similarity in code space instead of signal space (ablation) | VQScore · VoiceBank-DEMAND noisy test | correlation with PESQ: 0.6514 -> 0.7941 | Helped |
| use cosine similarity in code space rather than cosine similarity in signal space (ablation) | VQScore · IUB_cosine | correlation with real quality scores: 0.1819 -> 0.4880 | Helped |
| use cosine similarity in code space rather than L2 distance in code space (ablation) | VQScore · IUB_voices | correlation with real quality scores: -0.8436 -> 0.8604 | Helped |
| add adversarial training to the clean-speech-trained enhancement model (ablation) | VQ-VAE speech enhancement · VoiceBank-DEMAND noisy test | PESQ: 2.20 -> 2.38 (+0.18) | Helped |
| add adversarial training to the clean-speech-trained enhancement model (ablation) | VQ-VAE speech enhancement · DNS1 Real | OVRL: 2.589 -> 2.750 | Helped |
| use adversarial rather than Gaussian noise during training (ablation) | VQ-VAE speech enhancement · DNS1 Real | OVRL: 2.673 -> 2.750 | Helped |
| remove the adversarial cross-entropy training step (ablation) | VQ-VAE speech enhancement · VoiceBank-DEMAND noisy test | OVRL: 2.876 -> 2.797 | Hurt |
| remove the vector quantization module (ablation) | VQ-VAE speech enhancement · VoiceBank-DEMAND noisy test | BAK: 3.725 -> 3.212 | Hurt |
| remove mean removal from the encoder (ablation) | VQ-VAE speech enhancement · VoiceBank-DEMAND noisy test | code accuracy: 0.3878 -> 0.2449 | Hurt |
| remove vector quantization from clean-speech reconstruction (ablation) | VQ-VAE speech enhancement · VoiceBank-DEMAND clean test input | PESQ: 2.61 -> 3.91 | Helped |
| increase the quality-estimation codebook dimension from 16 to 32 (ablation) | VQScore · DNSMOS OVRL correlation validation | LCC: 0.7668 -> 0.8386 | Helped |
| vary the codebook size around (2048, 32) (ablation) | VQScore · DNSMOS OVRL correlation validation | LCC: 0.8332; 0.8386; 0.8317; 0.8297 | No clear effect |
Surprising findings
- VQScore computed with L2 distance is negatively or weakly correlated with real quality, whereas cosine similarity in code space works well.
- Enhanced speech from the VQ-based model has lower PESQ and higher speech distortion, while DNSMOS scores remain high.
Limitations
- Speech distortion (lower SIG) caused by finite discrete token combinations in the VQ module.
- Weak correlation on the VoiceMOS 2022 TTS test set (LCC about 0.46), below SpeechLMScore (LSTM)+rep (0.582).
- Validation/early stopping uses DNSMOS OVRL as a proxy, so some supervision enters model selection.
Experimental setup
Datasets: LibriSpeech clean 460h VoiceBank-DEMAND DNS1 test set DNS3 test set Conferencing Speech 2022 test clips (IUB_cosine, IUB_voices, Tencent) VoiceMOS 2022
Benchmarks: VoiceBank-DEMAND noisy test set DNS1 test set DNS3 test set Conferencing Speech 2022 test clips VoiceMOS 2022
Metrics: linear correlation coefficient PESQ DNSMOS SIG/BAK/OVRL STOI WER subjective ITU-T P.835 listening test
Baselines: DNSMOS P.835 TorchaudioSquim SpeechLMScore CNN-Transformer Demucs NyTT MetricGAN-U MMSE Wiener filter
Builds on: VQ-VAE SoundStream SpeechLMScore self-distillation PGD-style adversarial training
Closest papers by meaning
- Rethinking Training Targets, Architectures and Data Quality for Universal Speech Enhancement (NeurIPS 2026 · similarity 0.85)
- FINALLY: fast and universal speech enhancement with studio-like quality (NeurIPS 2024 · similarity 0.84)
- SCOREQ: Speech Quality Assessment with Contrastive Regression (NeurIPS 2024 · similarity 0.84)
- GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling (ICLR 2025 · similarity 0.83)
- Audio Large Language Models Can Be Descriptive Speech Quality Evaluators (ICLR 2025 · similarity 0.82)
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.