Research map / Large language models / AI and society
The State of Data Curation at NeurIPS: An Assessment of Dataset Development Practices in the Datasets and Benchmarks Track
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper builds a 18-element data-curation-based rubric and toolkit for dataset documentation and uses it to manually audit 60 NeurIPS Datasets and Benchmarks track papers (2021-2023), finding wide variation and weak documentation of environmental footprint, context awareness, and data management.
Problem
Dataset documentation quality in ML is inconsistent and often lacks data curation practices (ethics, environmental footprint, data management), and there is no systematic way to audit NeurIPS D&B dataset papers.
Approach
Translate data curation lifecycle and FAIR principles into a two-level (minimum standard pass/fail, excellence full/partial/none) rubric with a toolkit, iteratively refined with inter-rater reliability checks, and apply it to audit 60 NeurIPS D&B datasets.
Key results
- Inter-rater reliability rose across rounds; median ICC in final round was 0.90 for datasets, with category medians 0.83-0.98.
- Documentation quality varied widely: best dataset met 86% of minimum-standard criteria, worst 39%.
- Context awareness and environmental footprint had 0% minimum-standard pass rates; no evaluated dataset quantitatively assessed environmental footprint, and median minimum-standard pass rate declined from 78% (2021) to 61% (2023).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Iteratively refine the rubric and its guidance | Data curation evaluation framework · NeurIPS D&B dataset evaluations across rounds | inter-rater reliability | Helped |
| Apply the evaluation framework to the 2021-2023 NeurIPS D&B dataset sample | NeurIPS D&B datasets · Round 5 | documentation completeness: 86% ... 39% | Mixed |
| Document context awareness and positionality | NeurIPS D&B dataset documentation · Round 5 | context-awareness minimum-standard pass rate: 0% | No clear effect |
| Quantitatively assess dataset-development environmental footprint | NeurIPS D&B dataset documentation · Round 5 | environmental-footprint minimum-standard pass rate: 0% | No clear effect |
| Add further dataset documentation requirements over time | NeurIPS D&B track documentation practices · NeurIPS D&B datasets, 2021-2023 | median minimum-standard pass score per dataset: 78%, to 67%, to 61% | No clear effect |
| Add further dataset documentation requirements over time | NeurIPS D&B track documentation practices · NeurIPS D&B datasets, 2021-2023 | median full excellence score per dataset: 29%, 25%, and 13% | No clear effect |
| Use structured documentation | NeurIPS D&B dataset developers · NeurIPS D&B datasets, 2021-2023 | structured documentation rate: 80% in 2021 to 50% in 2023 | Mixed |
Surprising findings
- No measurable improvement in documentation quality across 2021-2023 despite increasingly stringent NeurIPS D&B requirements.
- Structured documentation use fell from 80% in 2021 to 50% in 2023.
Limitations
- Sample of 60 datasets may not be representative of the whole D&B track.
- Human, interpretive evaluations depend on reviewer judgment; disagreement review may introduce cross-dataset inconsistency.
- Documentation-based audit cannot capture the full complexity of actual data curation practice.
Experimental setup
Datasets: 60 NeurIPS Datasets and Benchmarks track papers (2021-2023)
Metrics: intra-class correlation (ICC) percentage of rubric criteria passed percentage of standard-of-excellence criteria fully met
Builds on: datasheets for datasets FAIR data principles digital curation lifecycle models
Closest papers by meaning
- A Taxonomy of Challenges to Curating Fair Datasets (NeurIPS 2024 · similarity 0.90)
- A Systematic Review of NeurIPS Dataset Management Practices (NeurIPS 2024 · similarity 0.90)
- Bridging the Data Provenance Gap Across Text, Speech, and Video (ICLR 2025 · similarity 0.86)
- Navigating Dataset Documentations in AI: A Large-Scale Analysis of Dataset Cards on HuggingFace (ICLR 2024 · similarity 0.86)
- Quality over Quantity: Boosting Data Efficiency Through Ensembled Multimodal Data Curation (AAAI 2025 · similarity 0.86)
- SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image Classification (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.