Research map / Large language models / Retrieval and knowledge
KoLA: Carefully Benchmarking World Knowledge of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds KoLA, a knowledge-centric LLM benchmark with a four-level cognitive taxonomy (19 tasks), known Wikipedia plus evolving seasonal data, and a standardized-score and self-contrast evaluation of 28 LLMs.
Problem
Existing LLM benchmarks are broad but shallow, use incomparable per-task metrics, and risk test-data leakage; there is no fair, depth-oriented benchmark for LLM world knowledge that also evaluates knowledge creation without heavy human evaluation.
Approach
Organize world-knowledge tasks into a Bloom-inspired four-level taxonomy (memorization, understanding, applying, creating), pair known Wikipedia data with continuously refreshed evolving data, report min-max standardized z-scores across models for comparability, and score knowledge creation with a self-contrast ROUGE-L metric comparing a model's free and knowledge-grounded completions.
Key results
- GPT-4 ranks first overall with an overall average standardized score of 2.33, ahead of GPT-3.5-turbo (1.34) and InstructGPT davinci v2 (1.12).
- Open-source models average a z-score of -0.29, below the overall average, and most open-source rankings declined between seasons 1 and 2.
- Self-contrast metric correlates 0.61 (Spearman) with human-judged faithfulness of created content; removing it lowers correlation with overall quality by 32%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase base-model size among models without alignment or instruction tuning | GPT-J and BLOOM model families · KoLA Knowledge Memorization ranking | Spearman correlation with model size: 0.79 | Helped |
| Apply instruction tuning | LLMs of different model sizes · KoLA Knowledge Applying | Spearman correlation between ability and model size: 0.02 to 0.53 | Helped |
| Apply alignment or instruction tuning | LLMs · KoLA Knowledge Memorization | correlation between model size and performance: 0.34 | Hurt |
| Remove the self-contrast metric from the overall knowledge-creating score (ablation) | KoLA knowledge-creating evaluation · Human-judged overall quality correlation | Spearman correlation: 32% decrease | Hurt |
| Use the self-contrast metric ∂(T,T_k) | KoLA knowledge-creating evaluation · Knowledge Creating tasks | Spearman correlation with human-rated faithfulness: 0.61 | Helped |
| Compare open-source models with commercial closed-source models | LLMs evaluated on KoLA · KoLA overall | average z-score for open-source models: -0.29 | Hurt |
Surprising findings
- Alignment (instruction tuning) raises correlation between model size and higher-level abilities but lowers the correlation with low-level knowledge memorization, an 'alignment tax' pattern.
- Evolving and known data performance are linearly correlated, but the gap between them is smaller for higher-level tasks.
Limitations
- Covers only 19 English tasks on concepts, entities and events, narrower than broad benchmarks.
- Automatic knowledge-creation evaluation depends on human reference content and may underrate novel but unaligned knowledge.
- Evolving data refresh requires ongoing annotation effort, limiting how many seasons and models can be added.
Experimental setup
Datasets: Wikidata5M Wikipedia MAVEN MAVEN-ERE COPEN FewNERD DocRED HotpotQA 2WikiMultihopQA MuSiQue KQA Pro KoRC
Benchmarks: KoLA
Metrics: F1 Accuracy Rouge-L standardized z-score Spearman correlation
Baselines: GPT-4 GPT-3.5-turbo Cohere-command LLaMa (65B) GLM (130B) FLAN-UL2 (20B) Tulu (7B) Vicuna (13B)
Builds on: LAMA Wikidata5M Bloom's taxonomy COPEN MAVEN
Closest papers by meaning
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.91)
- SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research (AAAI 2024 · similarity 0.90)
- PertEval: Unveiling Real Knowledge Capacity of LLMs with Knowledge-Invariant Perturbations (NeurIPS 2024 · similarity 0.90)
- KScope: A Framework for Characterizing the Knowledge Status of Language Models (NeurIPS 2025 · similarity 0.90)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.90)
- INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.