Research map / Large language models / AI and society
A Systematic Review of NeurIPS Dataset Management Practices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically reviews 238 NeurIPS Datasets and Benchmarks papers (2021-2023) on provenance, distribution, ethical disclosure, and licensing, finding inconsistent dataset management practices and calling for standardized data infrastructure.
Problem
Dataset management practices across the ML community are unclear and inconsistent, hindering diagnosis of provenance, licensing, hosting, and ethical issues in large training datasets.
Approach
Applies a qualitative, multi-aspect coding scheme (provenance, distribution, ethical disclosure, licensing) to all 2021 and 2022 and a random sample of 2023 NeurIPS dataset papers to map real-world dataset management practices at scale.
Key results
- 57% (136/238) of reviewed papers were post hoc datasets, with provenance often unclear due to ambiguous collection and curation descriptions.
- Hosting was dispersed: personal or lab websites (44), Zenodo (34), GitHub (33), Google Drive (32); only some sites like PhysioNet and Zenodo support metadata and versioning.
- CC BY was the most common license (61 papers); 36 papers had no explicit license information, and only about 40% of papers included ethical disclosures.
Surprising findings
- Datasheet usage was 48% in 2021, rose to 62% in 2022, and fell to 53% in 2023.
- Many dataset papers used software licenses (MIT, Apache, GPL) that legal scholars question for datasets.
Limitations
- Sample limited to NeurIPS Datasets and Benchmarks track, excluding main-track dataset papers.
- 2023 sampled (80 of 193 papers) and year-to-year comparisons are descriptive only given short time span.
- Review limited to four aspects and does not assess secondary impacts such as copyright violations or harmful content.
Experimental setup
Datasets: NeurIPS Datasets and Benchmarks track papers (2021-2023)
Metrics: percentage of papers per practice category hosting site counts license counts
Builds on: Data Provenance Explorer datasheets for datasets FAIR principles
Closest papers by meaning
- The State of Data Curation at NeurIPS: An Assessment of Dataset Development Practices in the Datasets and Benchmarks Track (NeurIPS 2024 · similarity 0.90)
- A Taxonomy of Challenges to Curating Fair Datasets (NeurIPS 2024 · similarity 0.87)
- Bridging the Data Provenance Gap Across Text, Speech, and Video (ICLR 2025 · similarity 0.85)
- Navigating Dataset Documentations in AI: A Large-Scale Analysis of Dataset Cards on HuggingFace (ICLR 2024 · similarity 0.84)
- Position: Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them? (ICML 2024 · similarity 0.84)
- Copycats: the many lives of a publicly available medical imaging dataset (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.