Research map / Large language models / Agents, code and math
Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HiST-LLM, a four-choice benchmark built from the Seshat Global History Databank to test LLM knowledge of global history, finding models beat chance (25%) but fall short of expert knowledge.
Problem
LLMs' graduate-level history knowledge is untested, and existing benchmarks are biased toward Western and recent history, so it is hard to measure balanced global historical knowledge.
Approach
Converts the expert-curated Seshat Global History Databank (36,000 data points across 600+ polities, with references and evidenced/inferred labels) into multiple-choice questions asking whether a trait was present, absent, inferred present, or inferred absent, and evaluates LLMs by region, time period, and variable category.
Key results
- Overall 4-choice balanced accuracy ranges from 33.6% (Llama-3.1-8B) to 46.0% (GPT-4-turbo), above the 25% chance level.
- Performance declines over time: GPT-4-turbo reaches 55.3% for 8000-6000 BCE but 38.7% for 1500-2000 CE.
- Models perform worst in Oceania and Sub-Saharan Africa; GPT-4-turbo leads in six of eight regions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4-turbo instead of Llama-3.1-8B (FP8) | HiST-LLM evaluation · HiST-LLM full dataset, 4-choice | balanced accuracy: 33.6 [33.2, 34.1] -> 46.0 [45.5, 46.5] | Helped |
| Use GPT-4-turbo instead of GPT-3.5-turbo | HiST-LLM evaluation · HiST-LLM full dataset, 4-choice | balanced accuracy: 35.6 [35.2, 35.9] -> 46.0 [45.5, 46.5] | Helped |
| Use GPT-4o instead of GPT-4-turbo | HiST-LLM evaluation · HiST-LLM full dataset, 2-choice | balanced accuracy: 60.0 [59.5, 60.6] -> 63.2 [62.7, 63.6] | Helped |
| Evaluate later rather than earlier historical periods | GPT-4-turbo · HiST-LLM temporal bins, 8k–6k BCE vs 1.5k–2k CE | balanced accuracy: 55.3 [48.0, 62.5] -> 38.7 [37.4, 40.0] | Hurt |
| Evaluate Oceania rather than Latin America | GPT-4-turbo · HiST-LLM geographic regions | balanced accuracy: 49.2 [47.0, 51.3] -> 38.0 [35.1, 40.8] | Hurt |
Surprising findings
- Models perform better on earlier historical periods than recent ones, despite more data existing for recent periods.
- Europe is not the region where models perform best; performance is highest in the Americas.
Limitations
- Seshat data is mostly drawn from English-language sources, limiting coverage of non-English regions.
- Benchmark reflects only current understanding; new evidence can change ground truth.
- Only one prompt format and temperature 0 were tested; models are a non-exhaustive selection.
Experimental setup
Datasets: Seshat Global History Databank
Benchmarks: HiST-LLM
Metrics: balanced accuracy adjusted balanced accuracy 2-choice balanced accuracy
Baselines: GPT-3.5-turbo Gemini-1.5-flash Llama-3-70B Llama-3.1-8B
Builds on: Seshat Global History Databank MMLU
Closest papers by meaning
- ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains (ICLR 2025 · similarity 0.87)
- BertaQA: How Much Do Language Models Know About Local Culture? (NeurIPS 2024 · similarity 0.86)
- KoLA: Carefully Benchmarking World Knowledge of Large Language Models (ICLR 2024 · similarity 0.85)
- HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models (ICLR 2026 · similarity 0.85)
- BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages (NeurIPS 2024 · similarity 0.85)
- Large Language Models are Geographically Biased (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.