Research map / Large language models / Agents, code and math
BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BenchmarkCards, a standardized documentation template for LLM benchmarks, auto-generating 100 card drafts with gpt-4o-mini and validating them through user interviews (10 users) and author feedback (25+ responses).
Problem
LLM benchmarks lack standardized documentation of objectives, data sources, methodology, metrics, and limitations, making selection, comparison, and interpretation difficult and prone to misuse.
Approach
A structured card template (benchmark details, purpose and users, data, methodology, targeted risks mapped to IBM AI Risk Atlas, ethical and legal considerations) for documenting LLM benchmarks, with cards generated from publications by an LLM and checked by benchmark authors and users.
Key results
- Auto-generated cards for 100 existing benchmarks; 25+ benchmark authors gave feedback, with 10 saying all information was correct or needed only minor clarifications.
- In a 10-participant user study, users praised conciseness and structure, and repeatedly requested concrete input/output examples and explicit comparisons between similar benchmarks (more than half of participants).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use standardized benchmark cards documenting objectives, methodology, data, and limitations | benchmark selection and understanding · User-Focused Study | perceived clarity | Helped |
| use standardized benchmark cards with comparable fields such as Targeted Risks | benchmark comparison and selection · User-Focused Study | benchmark comparability | Helped |
| generate benchmark cards from benchmark publications | benchmark authors reviewing their cards · Author-Focused Study | perceived accuracy and clarity | Helped |
Surprising findings
- Authors frequently flagged generated Targeted Risks as inapplicable or needing more context.
Limitations
- Long-term maintenance of cards in a decentralized ecosystem depends on community participation that cannot be guaranteed.
- Detailed limitation documentation could be exploited by malicious actors to game evaluations.
- Validation is based on a small qualitative user study (10 users) and author feedback rather than quantitative measures.
Experimental setup
Datasets: BBQ RealToxicityPrompts
Metrics: qualitative user feedback author-reported correctness and completeness
Builds on: Model Cards Datasheets for Datasets FactSheets Unitxt Cards
Closest papers by meaning
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.93)
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.92)
- LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.91)
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.91)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.90)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.