Research map / Large language models / Instruction and fine-tuning
Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Translates Winogrande and three MMLU clinical sections into 8 African languages, measures LLM performance gaps, and tests fine-tuning (LLM-as-annotator quality tiers, cross-lingual transfer, cultural appropriateness) with Llama 3 70B.
Problem
LLMs perform substantially worse in native African languages, largely because few reasoning or domain benchmarks exist there, so the size of the gap is unknown and methods to close it are untested.
Approach
Creates human-translated, validated African-language versions of Winogrande and MMLU clinical sections, quantifies SOTA LLM gaps, and compares fine-tuning strategies (mono- vs cross-lingual data, GPT-4o-scored data quality tertiles, data volume, and culturally appropriate subsets) using Llama 3 70B with LoRA.
Key results
- GPT-4o averages 12.0%-19.9% absolute accuracy gap between English and the 11 African languages across benchmarks.
- Mono-lingual fine-tuning of Llama 3 70B yields an average 5.6% gain, with high-quality data beating low-quality data by 5.4% on average.
- Cross-lingual transfer gives an average 2.9% gain; GPT-4o scores 3.0-3.3% higher on culturally appropriate Winogrande subsets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on examples in the evaluation language | Llama 3 70B · African-language benchmarks, average across languages and benchmarks | accuracy gain: +5.6% | Helped |
| Fine-tune on MMLU College Medicine examples | Llama 3 70B · MMLU Clinical Knowledge | accuracy gain: +17.4% on average | Helped |
| Fine-tune on MMLU College Medicine examples | Llama 3 70B · Belebele | accuracy gain: +9.4% on average | Helped |
| Fine-tune using data from another language | Llama 3 70B · African-language benchmarks, cross-lingual evaluations | accuracy gain: +2.9% on average | Helped |
| Select high-quality fine-tuning examples using GPT-4o quality scores instead of low-quality examples | Llama 3 70B · African-language benchmarks | accuracy: +5.4% on average | Helped |
| Double fine-tuning data from 33 to 66 samples | Llama 3 70B · MMLU Clinical Knowledge | accuracy gain: +2.3% on average | Helped |
| Evaluate culturally appropriate rather than inappropriate Winogrande questions | GPT-4o · Winogrande, African languages | accuracy: +3.3% average difference | Helped |
| Backtranslate African-language questions into English before evaluation | English-centric LLMs such as Llama 3 and Phi 3 · African-language benchmark questions | accuracy: 20%-30%; upwards of 40% for Belebele, MMLU College Medicine, and MMLU Clinical Knowledge | Helped |
| Use machine-translated rather than human-translated Winogrande text | State-of-the-art LLMs · Winogrande | accuracy: 0-3% | No clear effect |
| Backtranslate African-language benchmark text into English | GPT-4 and GPT-4o · Most languages and benchmarks | accuracy: 0%-10% | Mixed |
| Use machine-translated instead of human-translated text | State-of-the-art LLMs · Some African-language benchmark settings, including Igbo | accuracy | Mixed |
Surprising findings
- Machine translation into African languages sometimes performed as well as or better than human translation for some models and benchmarks, e.g., Igbo on MMLU.
- Inter-annotator agreement on cultural appropriateness was very low (Cohen's Kappa mostly near zero).
Limitations
- Fine-tuning only with Llama 3 70B, which still trails GPT-4o out of the box; fine-tuning done per language in isolation.
- Translator availability and dialect variability limited translation quality; human translation is costly to scale.
- Translated benchmarks may not reflect African-language-specific use cases.
Experimental setup
Datasets: Winogrande MMLU college medicine MMLU clinical knowledge MMLU virology Belebele Afrikaans, Zulu, Xhosa translations (pre-existing)
Benchmarks: Winogrande MMLU college medicine MMLU clinical knowledge MMLU virology Belebele
Metrics: accuracy performance gap ROUGE-1
Baselines: GPT-4o GPT-4 GPT-3.5 Llama 3 70B Llama 3 8B Phi 3 Mini Aya 23 Aya 101 BLOOMZ 7b1
Builds on: Winogrande MMLU Belebele LLM-as-a-Judge
Closest papers by meaning
- WAXAL: A Large-Scale Multilingual African Language Speech Corpus (NeurIPS 2026 · similarity 0.89)
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.88)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.88)
- Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages (ICLR 2025 · similarity 0.88)
- BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages (NeurIPS 2024 · similarity 0.88)
- Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.