Research map / Adversarial robustness and security / Certified and empirical robustness
Speech Robust Bench: A Robustness Benchmark For Speech Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Speech Robust Bench (SRB), a benchmark evaluating ASR robustness to common, semantic, speaker-attribute and adversarial corruptions at four severities, with ER/ERD/CE/CED and EDV metrics, on LibriSpeech test-clean and multilingual Spanish LibriSpeech.
Problem
ASR models lack systematic robustness evaluation under diverse real-world and adversarial audio corruptions, with no unified measure of both utility and prediction stability.
Approach
Adapts image robustness metrics (corruption error, error-degradation, variance) to sequence transcription by using word/character edit-based error rates normalized by a baseline ASR model, plus an error-degradation variance metric for stability across noise samples.
Key results
- Whisper Large achieves the top robustness on all four aggregate metrics (CWE 0.37, CWED 0.21, CCE 0.43, CCED 0.30), while Wav2Vec2 Base has CE/CED of 1.00 and Whisper Tiny 1.15/1.01 at word level.
- Larger models trained on more data are more robust to non-adversarial corruptions but show similar vulnerability to adversarial perturbations.
- On Spanish, mms-1b-fl102 is reported as the most robust model overall, with whisper-large-v2 having much higher CCED (2.53) due to RIR and voice conversion-like failures.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the larger Whisper model instead of Whisper tiny.en | Whisper ASR on LibriSpeech test-clean · LibriSpeech test-clean, averaged across corruptions and severities | CCE: 1.32 -> 0.43 | Helped |
| Use the larger Whisper model instead of Whisper tiny.en | Whisper ASR on LibriSpeech test-clean · LibriSpeech test-clean, averaged across corruptions and severities | CER: 0.40 -> 0.08 | Helped |
| Increase model size and training data | ASR models on non-adversarial perturbations · LibriSpeech test-clean | robustness | Helped |
| Increase model size and training data | ASR models under adversarial perturbations · LibriSpeech test-clean | robustness | No clear effect |
| Use MMS 1B instead of Whisper large-v2 | Spanish ASR · Multilingual LibriSpeech Spanish test | CCE: 0.91 -> 0.35 | Helped |
| Use MMS 1B instead of Whisper large-v2 | Spanish ASR · Multilingual LibriSpeech Spanish test | CER: 0.26 -> 0.20 | Helped |
| Transcribe Spanish rather than English with multilingual Whisper models | Whisper multilingual models · English and Spanish speech under corruptions | CER | Hurt |
| Use Whisper large-v2 instead of Whisper tiny | Whisper multilingual ASR on non-English speech · Spanish speech, speaker-gender disparity | CER disparity | Mixed |
Surprising findings
- Whisper Large has a large error spike under voice conversion severity 3 (Indian accent), with ER over three times Whisper Tiny's on that severity.
- Whisper Large is largely invariant to RIR on English but its CER spikes sharply under RIR on Spanish.
Limitations
- Baseline model choice (whisper-tiny.en vs. wav2vec2-base-960h vs. whisper-tiny) changes across sections, making CE/CED comparisons inconsistent.
- Paper draft contains many unresolved comments and incomplete sections (e.g., empty introduction, missing dataset table values), limiting clarity of the evaluation.
- Only 9 corruption types at 4 severities are evaluated on a single clean test subset of LibriSpeech, with a small set of models.
Experimental setup
Datasets: LibriSpeech test-clean LibriSpeech dev (500 utterances for universal attack) MS-SNSD Multilingual LibriSpeech Spanish simulated room impulse responses
Benchmarks: Speech Robust Bench (LibriSpeech test-clean) Multilingual LibriSpeech Spanish subset
Metrics: WER CER ERD CE CED EDV CEDV
Baselines: whisper-large-v2 whisper-tiny wav2vec2-base-960h hubert-large-ls960-ft wav2vec2-large-960h-lv60-self wav2vec2-large-robust-ft-libri-960h deepspeech
Builds on: ImageNet-C corruption robustness metrics PGD adversarial attack universal adversarial perturbations
Closest papers by meaning
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.88)
- BIGOS V2 Benchmark for Polish ASR: Curated Datasets and Tools for Reproducible Evaluation (NeurIPS 2024 · similarity 0.87)
- What Was That Again? Certified Robustness for Automatic Speech Recognition (NeurIPS 2026 · similarity 0.87)
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.86)
- Certification of Speaker Recognition Models to Additive Perturbations (AAAI 2025 · similarity 0.86)
- Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.