Research map / Large language models / AI and society

AI Evaluation Should Require Standardized Item-Level Data Releases

NeurIPS 2026 · Large language models › AI and society · approach: Social-science view of AI evaluation · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position paper arguing standardized item-level benchmark response data should be the default for AI evaluation, illustrated by the OpenEval archive (10M responses, 155k items) and CTT/IFA analyses of MMLU, MMLU-Pro, and BabiQA.

Problem

AI benchmark results are reported as aggregate scores while item-level response data is discarded or unshared, which makes validity, item quality, construct alignment, contamination, and saturation hard to assess.

Approach

Argues that item-level response data under a unified item-centered schema should be a community norm released alongside aggregate scores, and demonstrates feasibility via the OpenEval repository and psychometric analyses (classical test theory, item factor analysis).

Key results

• OpenEval covers over 155K items and 10M item-level responses, with 11 to 111 evaluated models per dataset (70.3 on average).
• Many MMLU-Pro items have very low difficulty for 72 post-June 2024 models, and MMLU-Pro shows far fewer low or negative discrimination items than MMLU.
• SVD-based IFA on BabiQA Task 15 finds three item clusters explained by the reference answer (animal names), suggesting construct misalignment.

Surprising findings

• Item clusters on BabiQA Task 15 are explained by the reference answer (animal type), a construct-irrelevant cue rather than deduction ability.
• Some MMLU-Pro items still show poor discrimination even after expert review.

Limitations

• Illustrations rely mainly on CTT and IFA; IRT, DIF, and diagnostic models are not explored.
• Risk mitigation for contamination and open release is conceptual, without concrete access-control safeguards.
• Empirical analyses are illustrative rather than definitive; sub-construct interpretations are tentative.

Experimental setup

Datasets: MMLU MMLU-Pro BabiQA GPQA Omni-MATH HELM CulturalBench EmoBench

Benchmarks: MMLU MMLU-Pro BabiQA Task 15

Metrics: item difficulty index item discrimination (point-biserial/Pearson with rest-total factor loadings

Builds on: Classical Test Theory Item Response Theory Item Factor Analysis Generalized Low Rank Models

Closest papers by meaning

• Can We Trust Item Response Theory for AI Evaluation? (NeurIPS 2026 · similarity 0.91)
• EVALUATION CARDS: An Interpretive Layer for AI Evaluation Reporting (NeurIPS 2026 · similarity 0.91)
• Position: AI Evaluation Should Learn from How We Test Humans (ICML 2025 · similarity 0.88)
• Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregation (NeurIPS 2026 · similarity 0.87)
• No Free Delivery Service: Epistemic limits of passive data collection in complex social systems (NeurIPS 2024 · similarity 0.87)
• Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking (NeurIPS 2026 · similarity 0.87)