Research map / Large language models / Retrieval and knowledge
BertaQA: How Much Do Language Models Know About Local Culture?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BertaQA, a parallel English-Basque trivia benchmark split into local (Basque culture) and global questions, and shows LLMs lag on local knowledge while Basque continued pretraining (Latxa) transfers local knowledge into English.
Problem
LLM evaluations are dominated by global or anglocentric topics, leaving local cultural knowledge, here Basque culture, poorly measured.
Approach
A multiple-choice trivia dataset with matched local and global subsets, parallel in Basque and English, used to compare model performance on local versus global knowledge and to test whether continued pretraining in a low-resource language transfers knowledge to English.
Key results
- GPT-4 Turbo reaches 91.7% on the global subset but 72.2% on the local subset of the English version, with an average local-global gap of 26.66 points across models.
- Continued pretraining of Llama 2 70B in Basque (Latxa) raises English local accuracy from 49.15 to 62.61 while global accuracy drops from 77.68 to 73.62.
- Translate-test and self-translate help more on global questions than local ones; for Latxa both methods hurt local performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Continue pretraining Llama 2 70B in Basque | Llama 2 70B · BertaQA English local | accuracy: 49.15 -> 62.61 (+13.46 points) | Helped |
| Continue pretraining Llama 2 70B in Basque | Llama 2 70B · BertaQA English global | accuracy: 77.68 -> 73.62 (-4.1 points) | Hurt |
| Continue pretraining Llama 2 13B in Basque | Llama 2 13B · BertaQA English local | accuracy: 43.61 -> 56.60 | Helped |
| Continue pretraining Llama 2 7B in Basque | Llama 2 7B · BertaQA English local | accuracy: 41.54 -> 47.72 | Helped |
| Continue pretraining Llama 2 in Basque | Llama 2 family · BertaQA English global | accuracy: 64.34 -> 53.26; 70.36 -> 67.47; 77.68 -> 73.62 | Hurt |
| Query Latxa in Basque instead of English | Latxa 70B · BertaQA local | accuracy: 62.61 -> 64.85 | Helped |
| Query Latxa in Basque instead of English | Latxa 70B · BertaQA global | accuracy: 73.62 -> 72.24 | Hurt |
| Translate Basque input into English using NLLB-200 (ablation) | Llama 2 70B · BertaQA local | accuracy: +5.29 | Helped |
| Translate Basque input into English using NLLB-200 (ablation) | Llama 2 70B · BertaQA global | accuracy: +16.81 | Helped |
| Use self-translation from Basque into English (ablation) | Latxa 70B · BertaQA local | accuracy: -4.70 | Hurt |
| Use self-translation from Basque into English (ablation) | Latxa 70B · BertaQA global | accuracy: -1.76 | Hurt |
| Use translate-test or self-translate rather than direct Basque inference (ablation) | Gemma 7B · BertaQA local and global | accuracy: -0.17; +3.30; -0.17; +1.79 | Mixed |
Surprising findings
- Continued pretraining in Basque improves English-prompted accuracy on local Basque-culture questions, suggesting cross-lingual knowledge transfer from a low-resource to a high-resource language.
- Prior conclusions that translation-based methods and training on other languages help or harm English do not hold when evaluated on local topics.
Limitations
- Local subset covers only Basque culture, so generalization to other minority cultures is untested.
- English version is professionally translated, which may introduce translationese artifacts.
- Web search analysis found some questions have temporally changing or ambiguous answers.
Experimental setup
Datasets: BertaQA
Benchmarks: BertaQA
Metrics: accuracy local-global accuracy delta
Baselines: GPT-3.5 Turbo GPT-4 GPT-4 Turbo Claude 3 Opus Claude 3 Sonnet Claude 3 Haiku Llama 2 Llama 3 Mistral Mixtral Qwen 1.5 Yi
Builds on: Latxa Llama 2 LM Evaluation Harness NLLB-200
Closest papers by meaning
- CultureLLM: Incorporating Cultural Differences into Large Language Models (NeurIPS 2024 · similarity 0.88)
- BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages (NeurIPS 2024 · similarity 0.87)
- INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (ICLR 2025 · similarity 0.87)
- All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages (CVPR 2025 · similarity 0.86)
- KoLA: Carefully Benchmarking World Knowledge of Large Language Models (ICLR 2024 · similarity 0.86)
- Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.