Research map / Label-efficient and robust learning / Domain adaptation and generalization
Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STAR, a source-free unsupervised adaptation method for speech foundation models that uses a token-level STAR indicator (combining confidence and attentive scores) to re-weight pseudo-label finetuning, plus utterance-level filtering.
Problem
Adapting attention-based speech foundation models (e.g., Whisper) to noisy, accented, and specific target domains without ground-truth labels or source data, where pseudo-label quality is hard to assess because softmax confidence is unreliable.
Approach
A novel token-level quality indicator that integrates decoder self-attention aggregate scores (reliable but unstable) with softmax confidence (stable but unreliable) through conflict/consistency-aware sigmoid gating, used to re-weight the ASR cross-entropy loss during self-training; combined with utterance-level filtering via Gaussian weight perturbation.
Key results
- STAR reduces WER by an average of 13.5% relative across 14 target domains with Whisper-Large-V3, e.g., -24.9% on LS-FreeSound babble and -20.0% on CommonVoice African.
- STAR approaches the supervised upper bound on some domains, e.g., CHiME-4 dev-real 3.9 vs 3.2 real-label WER and LS-FreeSound car 2.5 vs 2.4.
- STAR achieves over 10% relative WER reduction on OWSM, Canary, Parakeet-TDT, and yields up to 30.7% reduction with SeamlessM4T-V2 on CHiME-4 dev-real; ST gains average over 1.2 BLEU on FLEURS X-to-En.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use STAR adaptation instead of attentive-score token reweighting | Whisper-Large-V3 · 14 ASR target domains | WER (%): 6.2 -> 6.0; 9.5 -> 9.4; 4.1 -> 3.9; 6.6 -> 6.4; 31.3 -> 30.2; 15.0 -> 14.8; 2.6 -> 2.5; 44.9 -> 44.6; 5.0 -> 4.8; 5.2 -> | Helped |
| add utterance-level filtering to STAR (ablation) | STAR with Whisper-Large-V3 · CHiME-4 test-real | WER (%): 6.2 -> 6.0 | Helped |
| replace Gaussian disturbance filtering with beam search decoding (ablation) | utterance-level filtering · CHiME-4 test-real | WER (%): 6.4 -> 6.6 | Hurt |
| replace Gaussian disturbance filtering with consensus decoding (ablation) | utterance-level filtering · CHiME-4 test-real | WER (%): 6.4 -> 6.6 | Hurt |
| replace confidence token scores with attentive scores (ablation) | token-level reweighting · ASR target domains | WER (%) | Helped |
| use confidence-only token reweighting rather than STAR (ablation) | Whisper-Large-V3 · ASR target domains | WER (%) | Hurt |
| use attentive-score-only token reweighting rather than STAR (ablation) | Whisper-Large-V3 · ASR target domains | WER (%) | Hurt |
| freeze part of the model parameters during STAR finetuning (ablation) | STAR · CHiME-4 test-real | WER (%) | Hurt |
| use LoRA tuning instead of full finetuning (ablation) | STAR · CHiME-4 test-real | WER (%) | No clear effect |
| repeat pseudo-labeling and STAR adaptation for multiple iterations (ablation) | STAR · CHiME-4 test-real, Whisper large-v3 | WER (%): 6.0 -> 5.7 | Helped |
Surprising findings
- Adapting to CHiME-4 with STAR improves performance on other unrelated domains rather than causing catastrophic forgetting, without recalling source data.
- Softmax confidence is unreliable for Whisper: 52% of correct tokens get low confidence and 60% of wrong tokens get high confidence.
Limitations
- Relies on large, universally robust speech foundation models as pseudo-labelers; smaller domain-specific models (WavLM-Conformer, RNN-T) are not suitable.
- Gains are limited on colloquial and challenging data (SwitchBoard, CORAAL, RATS), where pseudo labels are poor and improvements mainly reflect function-word adjustments.
- Requires multiple decoding passes with weight perturbation for utterance-level filtering, adding compute; several hyperparameters (lambda, tau, alpha) are set empirically.
Experimental setup
Datasets: CHiME-4 LibriSpeech-FreeSound RATS CommonVoice 5.1 TED-LIUM 3 SwitchBoard LRS2 ATIS CORAAL FLEURS
Benchmarks: CHiME-4 LibriSpeech-FreeSound RATS CommonVoice TED-LIUM 3 SwitchBoard LRS2 ATIS CORAAL FLEURS X-to-En
Metrics: word error rate BLEU normalized cross-entropy
Baselines: Whisper (frozen) Whisper (self-train.) Whisper (real label) UTT_filter TOK_reweight (confidence) TOK_reweight (attentive) beam search filtering consensus decoding filtering
Builds on: Whisper self-training focal loss Monte Carlo dropout consensus decoding
Closest papers by meaning
- Master-ASR: Achieving Multilingual Scalability and Low-Resource Adaptation in ASR with Modular Learning (ICML 2023 · similarity 0.89)
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.89)
- In-Situ Text-Only Adaptation of Speech Models with Low-Overhead Speech Imputations (ICLR 2023 · similarity 0.89)
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.88)
- UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation (ICLR 2025 · similarity 0.86)
- E-BATS: Efficient Backpropagation-Free Test-Time Adaptation for Speech Foundation Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.