Research map / Large language models / Language, safety and interpretability
Measuring what Matters: Construct Validity in Large Language Model Benchmarks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic review of 445 LLM benchmarks from ICML/ICLR/NeurIPS and ACL venues, coded by 29 experts for construct validity, yielding eight recommendations and a checklist for building valid LLM benchmarks.
Problem
LLM benchmarks often fail to measure the phenomena they claim to measure (low construct validity), due to poorly defined phenomena, unrepresentative tasks, weak metrics and unsupported claims.
Approach
Applies psychometric construct validity concepts (face, content, ecological, predictive, convergent, discriminant validity) to a structured codebook and expert review of published LLM benchmarks, then distills the gaps into an actionable checklist of eight recommendations.
Key results
- Of 445 reviewed benchmarks, 78.2% define their phenomenon, but 47.8% of those definitions are contested.
- Only 16.0% of benchmarks conducted any statistical testing; 53.4% presented evidence for construct validity.
- 27.0% of benchmarks incorporated convenience sampling and 12.3% used only readily accessible datasets.
- Mean Brennan-Prediger Kappa of .524 across 30 categorical questions in a double-reviewed sample of 46 papers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Measure composite phenomenon sub-components separately | LLM benchmark design · LLM benchmarks | clarity and interpretation | Helped |
| Use representative sampling rather than relying on convenience samples | LLM benchmark datasets · Task item population | construct validity | Helped |
| Test known LLM weaknesses with prompt variations | GSM8K · GSM8K | robustness of results | Helped |
| Add contamination checks and held-out task items | GSM8K · GSM8K | validity of results | Helped |
| Benchmarking on repeated public items over time | GSM8K · GSM8K | scores over time | Mixed |
Surprising findings
- Nearly every reviewed paper had weaknesses in at least one construct validity area despite about half discussing validity.
- Only 7.7% of benchmarks sourced tasks solely from another benchmark, while 33.6% relied on a single source overall.
Limitations
- Restricted to leading conference proceedings, excluding industry or non-peer-reviewed benchmarks.
- GPT-4o mini screening may have introduced undetected false negatives.
- Limited reviewers per paper reduced robustness of individual reviews; moderate inter-rater agreement.
Experimental setup
Datasets: ICML, ICLR, NeurIPS proceedings 2018-2024 ACL, NAACL, EMNLP proceedings 2020-2024
Metrics: Brennan-Prediger Kappa F1 score (LLM screening vs human labels) percentage of benchmarks meeting practices
Builds on: BetterBench Jacobs and Wallach construct validity framework Raji et al. benchmark definition
Closest papers by meaning
- Benchmarking LLMs via Uncertainty Quantification (NeurIPS 2024 · similarity 0.92)
- BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.92)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.92)
- LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.92)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.92)
- An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.