Research map / Architectures and efficiency / RNNs and normalization
Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research
Bernard Koch, Emily Denton, Alex Hanna, Jacob Gates Foster
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Dataset usage became more concentrated over time | PWC task communities · 2015-2020 | Gini: 0.113 increase; median Gini 0.60 in 2020 | Mixed |
| NLP task communities had a lower rate of increasing concentration | PWC task communities · NLP tasks, 2015-2020 | Gini trend | Mixed |
| Task communities adopted datasets originating in other tasks | PWC parent tasks · Full sample and task categories | adoption proportion: 57.8%; Computer Vision 71.9%; Methodology 74.1%; NLP 27.4% | Mixed |
| Task communities created datasets specifically for their own task | PWC parent tasks · Full sample and task categories | creation proportion: 62.5%; Computer Vision 53.3%; Methodology 52.6%; NLP 76.0% | Mixed |
| Model dataset adoption and creation proportions over time | PWC task communities · 2015-2020 | adoption and creation proportions | No clear effect |
| Dataset usage was concentrated among a small set of introducing institutions | PWC dataset usages · June 2021 | share of usages: over 50% attributed to twelve institutions | Mixed |
| Dataset and institutional usage concentration increased | PWC dataset usages · Recent years | Gini: over 0.80 | Mixed |
| Use Pielou evenness instead of Gini (ablation) | Concentration analysis in task communities · PWC task communities over time | concentration trend: median Pielou evenness 0.69 | No clear effect |
| Include all parent tasks instead of only the 133 largest (ablation) | PWC task communities · Analysis 1 | Gini trend: median Gini 0.72 -> 0.65 | No clear effect |
| Include all parent tasks instead of only the 133 largest (ablation) | PWC task communities · Analysis 2 | median adoption proportion: 57.7% -> 69.7% | Mixed |
| Include all 1,025 tasks, including parent and child tasks (ablation) | PWC task communities · Analysis 2 | median adoption proportion: median creation 33%; median adoption 100% | Mixed |
| Relax the requirement for dataset and using-paper task labels to overlap (ablation) | PWC-wide concentration analysis · 2011-2020 | Gini: datasets 0.31 -> 0.86; institutions 0.38 -> 0.80 | No clear effect |
Closest papers by meaning
- Benchmark Data Repositories for Better Benchmarking (NeurIPS 2024 · similarity 0.90)
- The State of Data Curation at NeurIPS: An Assessment of Dataset Development Practices in the Datasets and Benchmarks Track (NeurIPS 2024 · similarity 0.89)
- A Taxonomy of Challenges to Curating Fair Datasets (NeurIPS 2024 · similarity 0.85)
- Position: Measure Dataset Diversity, Don't Just Claim It (ICML 2024 · similarity 0.85)
- No Metric to Rule Them All: Toward Principled Evaluations of Graph-Learning Datasets (ICML 2025 · similarity 0.85)
- A Systematic Review of NeurIPS Dataset Management Practices (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
