Research map / Large language models / AI and society
EVALUATION CARDS: An Interpretive Layer for AI Evaluation Reporting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Evaluation Cards, a unified reporting layer that composes benchmark metadata, run data, and model metadata into interpretive signals (reproducibility, completeness, provenance, comparability) rendered via reader modes, and audits 5,816 models and 101,955 results.
Problem
AI evaluation results are reported inconsistently across leaderboards, model cards, benchmark papers, and blogs, so readers cannot compare results, see what is omitted, or trace aggregate claims to evidence.
Approach
A unified, operational reporting layer that joins EEE run data, Auto-BenchmarkCards benchmark metadata, and model metadata into a five-level rollout hierarchy (family, composite, benchmark, split, metric), computes four interpretive signals over each result, and renders them through research and policy reader modes.
Key results
- 96.5% of 50,461 (model, benchmark, metric-path) triples lack at least one minimal reproducibility field.
- Median per-benchmark documentation completeness is 10.7% across 635 benchmarks.
- 98.2% of 49,865 (model, benchmark) pairs are reported by only one party; 51.9% of 181 multi-organization metric groups exceed the 5% divergence threshold.
Surprising findings
- First-party rows populate 0.0% of base reproducibility fields on paired reports versus 16.6% for third-party rows.
- Fields for raw score reporting are fully populated while benchmark-card documentation and provenance fields are largely absent.
Limitations
- Relies on source completeness; EEE is not a census of public reporting and Auto-BenchmarkCards validates accuracy but not comprehensiveness.
- Completeness measures documentation adequacy, not evaluation rigor; risk of safetywashing.
- Corpus overrepresents English-language benchmarks and frontier-scale models; interviews mainly North American.
Experimental setup
Datasets: Every Eval Ever (EEE) Auto-BenchmarkCards hub-stats models.dev
Benchmarks: MMLU-Pro MATH-500
Metrics: reproducibility gap rate documentation completeness cross-party divergence rate entity resolution accuracy
Baselines: BenchmarkCards HELM OpenLLM Leaderboard Inspect BetterBench
Builds on: Auto-BenchmarkCards Every Eval Ever Model Cards best fit framework synthesis
Closest papers by meaning
- AI Evaluation Should Require Standardized Item-Level Data Releases (NeurIPS 2026 · similarity 0.91)
- Models That Know How Evaluations Are Designed Score Safer (NeurIPS 2026 · similarity 0.88)
- Position: AI Evaluation Should Learn from How We Test Humans (ICML 2025 · similarity 0.86)
- Rethinking XAI Evaluation: A Human-Centered Audit of Shapley Benchmarks in High-Stakes Settings (NeurIPS 2026 · similarity 0.85)
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation (ICLR 2026 · similarity 0.85)
- Can We Trust Item Response Theory for AI Evaluation? (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.