Research map / Label-efficient and robust learning / Test-time adaptation
E-BATS: Efficient Backpropagation-Free Test-Time Adaptation for Speech Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes E-BATS, a backpropagation-free single-utterance test-time adaptation method for speech foundation models that combines CNN-level prompt adaptation optimized by CMA-ES, a multi-scale utterance/token alignment loss with entropy, and a test-time EMA over CMA-ES statistics.
Problem
Speech foundation models degrade under acoustic domain shifts (noise, accents), and existing TTA methods are either memory-intensive (backpropagation-based) or inaccurate when backpropagation-free, especially for single-utterance speech recognition.
Approach
Learn a prompt vector added to CNN encoder latent features using derivative-free CMA-ES guided by a multi-scale loss (blank-excluded entropy, utterance-level centroid alignment to source statistics, and confidence-weighted token-level alignment), and carry CMA-ES statistics across utterances via an exponential moving average.
Key results
- On Wav2Vec2ForCTC-Base, E-BATS averages 15.6% WER over Gaussian noise conditions versus 16.4-18.7% for the strongest BP-based methods (DSUTA, CEA, SUTA) and 24.8% for FOA among BP-free baselines (average over noise levels).
- Reports 2.0x-6.4x peak GPU memory savings compared with backpropagation-based methods and 4.1%-13.5% accuracy gains over BP-free baselines.
- On HuBERTForCTC-Large, E-BATS achieves 6.4% average WER over Gaussian noise, the best among BP-free methods and competitive with BP-based methods (CEA 5.8, SUTA 5.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use E-BATS instead of the strongest BP-free baseline | Wav2Vec2ForCTC-Base · LibriSpeech test-other Gaussian noise (Avg) | WER: 16.4 -> 15.6 | Helped |
| Use E-BATS instead of DSUTA | Wav2Vec2ForCTC-Base · CHiME-3 Mixed | WER: 24.1 -> 24.3 | No clear effect |
| Use E-BATS instead of the strongest BP-free baseline | HuBERTForCTC-Large · LibriSpeech test-other Gaussian noise (Avg) | WER: 6.1 -> 6.4 | Mixed |
| Use E-BATS instead of DSUTA | HuBERTForCTC-Large · CHiME-3 Single | WER: 13.3 -> 14.0 | Hurt |
| Adapt CNN features directly instead of adding prompts at transformer input (ablation) | E-BATS · CHiME-3 Single | WER: 34.2 -> 24.0 | Helped |
| Add token-level alignment loss (ablation) | E-BATS multi-scale loss with T-EMA · CHiME-3 Single | WER: 24.3 -> 24.0 | Helped |
| Add utterance-level alignment loss to entropy-only loss (ablation) | E-BATS loss without T-EMA · CHiME-3 Mixed | WER: 49.6 -> 25.5 | Helped |
| Use T-EMA rather than resetting CMA-ES each utterance (ablation) | E-BATS · CHiME-3 Mixed | WER: 26.5 -> 24.3 | Helped |
| Use T-EMA rather than continuous adaptation without reset (ablation) | E-BATS · CHiME-3 Mixed | WER: 25.4 -> 24.3 | Helped |
| Increase T-EMA decay to 0.99 rather than use 0.90 (ablation) | E-BATS · CHiME-3 cafe-real | WER: 37.9 -> 39.9 | Hurt |
| Increase CMA-ES population from 10 to 50 candidates (ablation) | E-BATS · CHiME-3 cafe-real | WER: 41.8 -> 37.9 | Helped |
| Add utterance-level loss to entropy-only objective (ablation) | E-BATS · CHiME-3 Mixed trivial predictions | problematic prediction percentage: 37.50% -> 0.61% | Helped |
Surprising findings
- Entropy minimization alone collapses to blank or single-character predictions on CHiME-3-Mix (37.5% of predictions trivial), and adding the utterance-level loss reduces this to 0.61%.
- Prompt injection at the CNN feature encoder outperforms conventional transformer-input prompt concatenation (24.0 vs 34.2 WER on CHiME-3 single).
Limitations
- Iterative CMA-ES optimization adds adaptation latency and does not fully exploit GPU parallelism.
- Experiments limited to CTC-based ASR with two backbones (Wav2Vec2 Base and HuBERT Large) and noise/accent shifts.
- Token-level loss relies on pseudo-labels whose reliability depends on prediction confidence.
Experimental setup
Datasets: LibriSpeech test-other CHiME-3 CHiME-3-Mix CommonVoice TEDLIUM-v2
Benchmarks: LibriSpeech test-other with Gaussian noise CHiME-3 CHiME-3-Mix CommonVoice TEDLIUM-v2
Metrics: word error rate peak GPU memory
Baselines: TENT SAR EATA CoTTA SUTA CEA SGEM DSUTA CSUTA AWMC T3A LAME
Builds on: CMA-ES SUTA FOA prompt tuning
Closest papers by meaning
- The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models (NeurIPS 2025 · similarity 0.87)
- Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignment (NeurIPS 2025 · similarity 0.86)
- BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models (NeurIPS 2024 · similarity 0.86)
- Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts (NeurIPS 2026 · similarity 0.86)
- Realistic Test-Time Adaptation of Vision-Language Models (CVPR 2025 · similarity 0.86)
- DELTA-TTS: Adapting Autoregressive Model into a Diffusion Language Model for Text-to-Speech (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.