Research map / Large language models / Retrieval and knowledge
INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds INCLUDE, a 197,243-question multiple-choice benchmark from local exams across 44 languages and 52 countries, and evaluates 15 LLMs with a regional-knowledge focus.
Problem
Multilingual LLM benchmarks are scarce and usually translated from English, so they miss regional and cultural knowledge of the languages' actual deployment environments.
Approach
Collect native-language academic, professional, and regional license exams from local sources instead of translating English benchmarks, and annotate exam sources by academic domain and regionality (agnostic, implicit, explicit, cultural).
Key results
- GPT-4o achieves the highest accuracy, about 77.1% (5-shot) on INCLUDE-Lite across all languages.
- Aya-expanse-32B outperforms its 8B counterpart by about 12% on average, and Qwen2.5-14B beats Qwen2.5-7B by about 7%.
- GPT-4o performs worse on cultural history exams than on region-implicit world history exams in almost every language (Telugu is the exception).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use zero-shot CoT instead of 5-shot prompting | GPT-4o · INCLUDE-lite | accuracy: 77.3 -> 79.0 | Helped |
| Use zero-shot CoT instead of 5-shot prompting | Llama-3.1-70B-Instruct · INCLUDE-lite | accuracy: 70.6 -> 60.6 | Hurt |
| Use zero-shot CoT instead of 5-shot prompting | Qwen2.5-14B · INCLUDE-lite | accuracy: 61.4 -> 47.3 | Hurt |
| Use English instructions instead of in-language instructions | models evaluated · INCLUDE-lite | accuracy: ~1.5% | Helped |
| Add a regional prefix to the prompt | models evaluated, except Aya-expanse-8B · INCLUDE-lite | accuracy | No clear effect |
| Increase the generation window from 50 to 512 tokens | GPT-4o · INCLUDE-lite | accuracy: +3.1% | Helped |
| Increase the model size from 8B to 32B | Aya-expanse · INCLUDE-lite | accuracy: ~12% | Helped |
| Increase the model size from 7B to 14B | Qwen2.5 · INCLUDE-lite | accuracy: ~7% | Helped |
| Apply instruction tuning | Gemma-7B · INCLUDE-lite | accuracy: ~15% | Hurt |
| Apply instruction tuning | models evaluated · INCLUDE-lite | accuracy | No clear effect |
| Evaluate on INCLUDE-tiny rather than INCLUDE-lite | models evaluated · INCLUDE-tiny and INCLUDE-lite | accuracy: within 1% | No clear effect |
| Use Harness-Eval rather than the paper's evaluation framework | models evaluated · INCLUDE-lite | accuracy: within ~1% | No clear effect |
Surprising findings
- Instruction-tuned models often perform on par with or worse than their base counterparts; Gemma-7B-Instruct drops about 15%.
- Zero-shot CoT lowers accuracy for Llama-3.1-70B-Instruct and Qwen2.5-14B, with the largest drops on math-heavy subjects.
Limitations
- Exam difficulty varies across languages and exam sources, which complicates cross-language comparison.
- Regionality labels are assigned at the exam-topic level rather than per question.
- Format errors can distort scores, especially for models that do not follow few-shot output formats in some languages.
Experimental setup
Datasets: INCLUDE (native-language exams) ArabicMMLU ChineseMMLU TurkishMMLU PersianMMLU VNHSGE EXAMS
Benchmarks: INCLUDE-Lite INCLUDE-Tiny
Metrics: accuracy answer accuracy format error rate
Baselines: Llama-3.1-70B-Instruct Aya-expanse-32B Qwen2.5-14B Aya-expanse-8B Mistral-7B Gemma-7B Qwen2.5-7B Llama-3.1-8B
Builds on: MMLU ArabicMMLU EXAMS Harness-Eval
Closest papers by meaning
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.90)
- XIFBench: Evaluating Large Language Models on Multilingual Instruction Following (NeurIPS 2025 · similarity 0.90)
- BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.90)
- KoLA: Carefully Benchmarking World Knowledge of Large Language Models (ICLR 2024 · similarity 0.90)
- LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.90)
- MindMerger: Efficiently Boosting LLM Reasoning in non-English Languages (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.