Research map / Learning theory and algorithms / Prediction and decision losses

Benchmark Data Repositories for Better Benchmarking

NeurIPS 2024 · Learning theory and algorithms › Prediction and decision losses · approach: Active learning and subset selection · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position paper defining "benchmark data repositories" and proposing considerations (dataset citations, connection metadata, licenses, quality review, deprecation, compositional/task metadata, uncertainty, living datasets) for how such repositories can improve ML benchmarking.

Problem

Benchmark datasets in ML are often poorly documented, lack persistent identifiers, carry licensing ambiguity, contain unaddressed issues, and are used in narrow, non-reproducible ways; the role of the data repositories hosting them has received little attention.

Approach

Introduces the term "benchmark data repository" and argues that repositories (e.g., HuggingFace Datasets, Kaggle, OpenML, Papers with Code, TFDS, UCI) can be designed to address dataset and evaluation criticisms through DOIs/citations, connection metadata, licenses, quality review, deprecation processes, compositional/task metadata, uncertainty reporting, and living datasets.

Key results

• Survey of over 100 computer vision dataset papers found 36.6% gave no description of human annotators and only 7.8% reported annotator demographics (cited, not new).
• In a survey of over 1800 text datasets, 69% had 'unspecified' licenses on HuggingFace (cited).
• Only 5 of 100 papers introducing human-subjects datasets mentioned an IRB or equivalent (cited).

Surprising findings

• Several popular repositories (e.g., Zenodo, Mendeley) view their role as only providing infrastructure and not conducting data review.
• The UCI BUPA Liver Disorders dataset's last column, an unexplained split indicator, was mistakenly treated as a class label by many highly-cited papers.

Limitations

• Recommendations are not empirically validated; no experiments are conducted.
• Incentives for repositories, dataset creators, and reviewers remain an open challenge.
• Authors explicitly refrain from taking a stance on whether repositories should include leaderboards.

Experimental setup

Builds on: Datasheets for Datasets Data Cards FAIR principles Croissant

Closest papers by meaning

• Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods (AAAI 2024 · similarity 0.86)
• How Benchmark Prediction from Fewer Data Misses the Mark (NeurIPS 2025 · similarity 0.85)
• The State of Data Curation at NeurIPS: An Assessment of Dataset Development Practices in the Datasets and Benchmarks Track (NeurIPS 2024 · similarity 0.85)
• No Metric to Rule Them All: Toward Principled Evaluations of Graph-Learning Datasets (ICML 2025 · similarity 0.84)
• Efficient Lifelong Model Evaluation in an Era of Rapid Progress (NeurIPS 2024 · similarity 0.84)
• Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining (NeurIPS 2025 · similarity 0.83)