Research map / Large language models / AI and society
Fit for our purpose, not yours: Benchmark for a low-resource, Indigenous language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Critiques FLoRes/FLEURS as unfit for the Māori language and presents Te Taumata, a community-curated, sub-benchmarked ASR benchmark for Māori with native-sound, contemporary, bilingual and NZ English subsets.
Problem
Popular multilingual benchmarks (FLoRes-101, FLEURS) are machine-translated or poorly curated, with errors, limited vocabulary and little cultural relevance, making them unfit for evaluating speech technology for the low-resource Indigenous Māori language.
Approach
A community-led, language-centric ASR benchmark built from the Te Hiku Media archive, with sub-benchmarks reflecting native sound, speaker diversity, codeswitching and aspirations of the language community, and with expert-written orthography guidelines and quality assurance.
Key results
- The in-house Māori ASR model's word error rate on Te Taumata dropped from 27% to 10% across iterations.
- Te Taumata contains 4,835 sentences, 89,277 words, 6,402 unique words and 37 speakers, versus FLEURS' 1,211 sentences, 30,794 words and 4 speakers.
- As the model improved on the Native Sound sub-benchmark, it progressively got worse on FLEURS, which the authors attribute to FLEURS lacking native-sound data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Improve the Te Hiku Media ASR across iterations | Te Hiku Media ASR · Māori ASR benchmark | WER: 27% to 10% | Helped |
| Improve the Te Hiku Media ASR across iterations | Te Hiku Media ASR · Te Taumata Native Sound | WER: 26.7 % -> 22.7 % | Helped |
| Prioritize Native Sound performance in ASR development | Te Hiku Media ASR · Te Taumata First Language Contemporary and Bilingual | WER | Hurt |
| Improve the Te Hiku Media ASR across iterations | Te Hiku Media ASR · FLEURS Māori | WER: 25.4 % -> 30.5 % | Hurt |
Surprising findings
- Model improvements on the native-speaker sub-benchmark coincided with degrading performance on FLEURS.
Limitations
- Benchmark is biased toward the far-north regional Māori dialect.
- The NZ English and bilingual sub-benchmark has a larger male representation due to data availability.
- Creating high-quality benchmarks requires substantial effort, which may be hard for other low-resource languages.
Experimental setup
Datasets: FLEURS Te Taumata (Te Hiku Media archive)
Benchmarks: Te Taumata FLEURS
Metrics: word error rate sentences per utterance words per utterance unique words
Baselines: Whisper-large-v2 Whisper-large-v3 Meta MMS Mozilla DeepSpeech
Builds on: FLEURS FLoRes-101 Conformer Whisper
Closest papers by meaning
- Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments (AAAI 2025 · similarity 0.87)
- Linguini: A benchmark for language-agnostic linguistic reasoning (NeurIPS 2025 · similarity 0.86)
- INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (ICLR 2025 · similarity 0.86)
- LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low Resource and Extinct Languages (NeurIPS 2024 · similarity 0.85)
- AI-Driven Multicultural Identity Preservation (AAAI 2025 · similarity 0.85)
- RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.