Research map / Large language models / AI and society
Position: Human Baselines in Model Evaluations Need Rigor and Transparency (With Recommendations & Reporting Checklist)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing human baselines in foundation model evaluations need more rigor and transparency; proposes a lifecycle framework and reporting checklist, and uses it to systematically review 115 human baselines.
Problem
Human baselines used to claim AI "super-human" performance are often small, biased, use different test sets or instruments than AI evaluations, and omit key methodological details, so human vs. AI comparisons are unreliable.
Approach
Draws on measurement theory and social-science survey methodology to derive recommendations across five stages (design, recruitment, execution, analysis, documentation) and a reporting checklist, then applies the checklist to audit 115 existing human baselines.
Key results
- Of 115 reviewed human baselines, 41% used different test sets for AI and human evaluation, and only 35% iteratively developed baseline instruments.
- Only 2% of reviewed baselines reported power analyses, and the median human sample size was 8 (mean 90).
- Only 37% of evaluations provided interval or distribution estimates and only 8% performed statistical tests; 88% showed UI differences between human and AI evaluation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the same representative test set for human and AI evaluation | human-versus-AI performance comparisons · foundation model evaluations | comparability of performance | Helped |
| iteratively develop and validate human baseline instruments | human baseline instruments · foundation model evaluations | instrument clarity and consistent interpretation | Helped |
| use a representative sampling strategy for selecting baseliners | human baseline samples · foundation model evaluations | external validity | Helped |
| apply quality controls during recruitment | human baseline recruitment · foundation model evaluations | data quality | Helped |
| apply quality controls during baseline execution | human baseline execution · foundation model evaluations | data quality | Helped |
| control method effects and use identical tasks for humans and AI | human-versus-AI evaluation · foundation model evaluations | internal validity | Helped |
| collect qualitative explanations from baseliners | human baseline analysis · foundation model evaluations | interpretability of human-AI performance differences | Helped |
| report uncertainty estimates for human-versus-AI performance differences | human baseline analysis · foundation model evaluations | assessment of underlying performance distributions | Helped |
| use consistent evaluation metrics, scoring methods, and rubrics across humans and AI | human-versus-AI performance comparisons · foundation model evaluations | fairness of comparisons | Helped |
| document baselining methods and baseliner information | human baseline reporting · foundation model evaluations | reproducibility and external assessment | Helped |
Surprising findings
- All reviewed baselines failed to report at least some checklist items.
- Only 14% of articles reported compliance with or exemption from ethics review.
Limitations
- Meta-review sample was purposively collected and may be biased.
- Scope limited to foundation model evaluations and primary human baseline data collection; does not cover LLM-as-judge or human uplift studies.
- Recommendations may be costly; maximal rigor is often infeasible for budget-constrained researchers.
Experimental setup
Datasets: 115 human baselines from academic publications and gray literature in foundation model evaluations
Metrics: percentage of reviewed baselines meeting each checklist item sample size statistical power analysis rate interval estimate and statistical test rates
Builds on: measurement theory (Zhao et al., 2025) Reuel et al. 2024 evaluation framework REFORMS consensus recommendations
Closest papers by meaning
- Position: AI Evaluation Should Learn from How We Test Humans (ICML 2025 · similarity 0.87)
- Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers (NeurIPS 2025 · similarity 0.86)
- Can We Trust Item Response Theory for AI Evaluation? (NeurIPS 2026 · similarity 0.86)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.85)
- AI Evaluation Should Require Standardized Item-Level Data Releases (NeurIPS 2026 · similarity 0.85)
- Position: Towards Bidirectional Human-AI Alignment (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.