Research map / Large language models / Retrieval and knowledge
BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BLEnD, a hand-crafted benchmark of 52.6k short-answer and multiple-choice everyday-culture questions across 16 countries/regions and 13 languages, showing LLMs perform worse on underrepresented cultures and low-resource local languages.
Problem
LLMs lack culture-specific everyday knowledge (food, sports, family, holidays, etc.) especially for underrepresented regions and non-English languages, and existing cultural benchmarks are single-language or drawn from Wikipedia/social media.
Approach
Human-crafted everyday-life cultural questions written and answered by native annotators from each country/region, with the same question set localized across 13 languages, evaluated via short-answer matching against annotator answers and multiple-choice questions with distractors drawn from other countries' answers.
Key results
- GPT-4 is the best-performing model, with a 57.34% difference between highest and lowest represented cultures in the short-answer format.
- Average SAQ performance is 79.22% for US culture in English versus 12.18% for Ethiopia in Amharic across all tested models.
- For low-resource-language cultures (e.g., Azerbaijani, Sundanese, Amharic), English prompting outperforms local-language prompting, while mid-to-high-resource cultures perform better in local languages.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate cultures represented by less-resourced languages rather than highly represented cultures | LLMs on BLEnD SAQ · US culture in English versus Ethiopian culture in Amharic | accuracy: 79.22% vs 12.18% | Hurt |
| Prompt in the local language instead of English for cultures represented by high-resource languages | LLMs on BLEnD SAQ · Cultures represented by Spanish and Chinese | accuracy | Helped |
| Prompt in English instead of the local language for cultures represented by low-resource languages | LLMs on BLEnD SAQ · Azerbaijani, Sundanese, and Amharic cultures | accuracy | Helped |
| Use multiple-choice questions rather than short-answer questions | LLMs on BLEnD · BLEnD MCQ vs SAQ | performance | Helped |
| Evaluate food and holidays/leisure questions rather than work-life or education questions | LLMs on BLEnD SAQ · Question categories | accuracy | Hurt |
| Use a region-centric model for its associated culture | Qwen1.5-72B and HyperCLOVA-X on BLEnD SAQ · Chinese and Korean cultural questions | accuracy | Helped |
| Prompt GPT-4 about South Korean rather than North Korean culture | GPT-4 on BLEnD SAQ · South Korea vs North Korea | accuracy: 31.63% | Helped |
| Prompt GPT-4 about Spanish rather than Mexican culture | GPT-4 on BLEnD SAQ · Spain vs Mexico | accuracy: 4.35% | Mixed |
Surprising findings
- For low-resource-language cultures, models perform better when prompted in English than in the local language.
- GPT-4 shows a 31.63% performance gap between South Korea and North Korea despite sharing the Korean language.
Limitations
- Relatively small number of annotators (typically five per question), sometimes from the same locality.
- Short-answer string-matching evaluation misses word variations and cannot assess long-form responses.
- Most language experts were English-proficient academics, which may bias construction; multilingual MCQ not yet available.
Experimental setup
Datasets: BLEnD
Benchmarks: BLEnD
Metrics: accuracy (short-answer match rate) multiple-choice accuracy Pearson correlation
Baselines: GPT-4 GPT-3.5 Claude-3-Opus Claude-3-Sonnet Claude-3-Haiku Llama-3.1-70B Gemini-1.0-Pro Qwen-1.5-72B
Builds on: GeoMLAMA CultureAtlas CLIcK
Closest papers by meaning
- Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures (NeurIPS 2026 · similarity 0.89)
- CultureLLM: Incorporating Cultural Differences into Large Language Models (NeurIPS 2024 · similarity 0.88)
- KoLA: Carefully Benchmarking World Knowledge of Large Language Models (ICLR 2024 · similarity 0.88)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.88)
- INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (ICLR 2025 · similarity 0.88)
- Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.