Research map / Large language models / Language, safety and interpretability
BIGOS V2 Benchmark for Polish ASR: Curated Datasets and Tools for Reproducible Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Curates 24 open Polish speech datasets into the BIGOS V2 and PELCRA for BIGOS benchmarks and evaluates 25 commercial and free ASR models with open evaluation tools and a public leaderboard.
Problem
Polish ASR evaluation is hampered by underused, hard-to-access speech datasets, no standard benchmark, and missing reproducible evaluation tooling.
Approach
A pipeline that surveys and catalogs public speech datasets, curates them into a standardized BIGOS-format benchmark with unified audio and metadata, and runs reproducible multi-scenario ASR evaluation with public dashboard and open tools.
Key results
- Whisper Large V3, Whisper Cloud and AssemblyAI best were the top 3 models by WER on BIGOS V2.
- Commercial systems had median WER of 12.96 vs 15.47 for free systems on read BIGOS V2 speech; on conversational PELCRA, 29.90 vs 34.18.
- Applying all normalization methods reduced WER by 15.22 p.p. on BIGOS V2 and 15.78 p.p. on PELCRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use commercial rather than free ASR systems | ASR systems evaluated on BIGOS V2 · BIGOS V2 | median WER: 15.47 -> 12.96 | Helped |
| Use commercial rather than free ASR systems | ASR systems evaluated on PELCRA · PELCRA | median WER: 34.18 -> 29.90 | Helped |
| Apply all normalization methods | ASR evaluation on BIGOS V2 · BIGOS V2 | WER: -14.63 p.p. | Helped |
| Apply lowercase normalization | ASR evaluation on BIGOS V2 · BIGOS V2 | WER reduction [p.p.]: -6.06 | Helped |
| Apply punctuation normalization | ASR evaluation on BIGOS V2 · BIGOS V2 | WER reduction [p.p.]: -7.61 | Helped |
| Increase model size | ASR models evaluated on BIGOS V2 and PELCRA · BIGOS V2 and PELCRA | WER | Helped |
| Use moderate rather than extreme speech rates | ASR systems evaluated on BIGOS V2 and PELCRA · BIGOS V2 and PELCRA | WER | Helped |
Surprising findings
- Normalization of references and hypotheses reduced WER by about 15 percentage points, showing lexical metrics are highly sensitive to formatting.
- Whisper hallucinated on unnaturally slow speech in the pwr-viu-unk subset, producing high WER despite the subset otherwise being easy.
Limitations
- Popular public datasets such as Common Voice and MLS may have been in the training data of evaluated systems, threatening validity.
- Test transcripts are not manually verified and no new non-public test recordings are included.
- Semantic metrics and manual annotations are not yet included.
Experimental setup
Datasets: Common Voice 15 Multilingual LibriSpeech (Polish) Clarin Studio Clarin Mobile PWR datasets M-AI Labs Polish AZON read and spontaneous Google FLEURS PolyAI Minds14 PolEval 22 Diabiz SpokesMix SpokesBiz
Benchmarks: BIGOS V2 PELCRA for BIGOS
Metrics: WER CER SER MER
Baselines: Google STT V1 Google STT V2 Azure STT AssemblyAI NeMo MMS Wav2Vec2
Builds on: BIGOS V1 Common Voice Multilingual LibriSpeech FLEURS Whisper
Closest papers by meaning
- Speech Robust Bench: A Robustness Benchmark For Speech Recognition (ICLR 2025 · similarity 0.87)
- WorldSpeech: A Multilingual Speech Corpus from Around the World (NeurIPS 2026 · similarity 0.87)
- WAXAL: A Large-Scale Multilingual African Language Speech Corpus (NeurIPS 2026 · similarity 0.86)
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration (AAAI 2025 · similarity 0.86)
- LAMA-UT: Language Agnostic Multilingual ASR Through Orthography Unification and Language-Specific Transliteration (AAAI 2025 · similarity 0.85)
- Recording for Eyes, Not Echoing to Ears: Contextualized Spoken-to-Written Conversion of ASR Transcripts (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.