Research map / Large language models / AI and society
Navigating Dataset Documentations in AI: A Large-Scale Analysis of Dataset Cards on HuggingFace
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Large-scale empirical analysis of 7,433 non-empty Hugging Face dataset cards, measuring adherence to the community template, section emphasis, topic modeling of limitation sections, a Usage-section effect, and human quality ratings.
Problem
Dataset documentation is recognized as essential for reliability, reproducibility and transparency of ML, yet there is little systematic empirical understanding of current documentation practices by dataset creators.
Approach
Uses the full population of Hugging Face dataset cards to measure how creators follow the community-endorsed dataset card template, how section length varies with downloads, and what topics appear in each section, and links documentation to popularity and human-perceived quality.
Key results
- 86.0% of the top 100 downloaded dataset cards fill all five community sections versus 7.9% of cards with no downloads.
- Dataset Description and Dataset Structure make up 36.2% and 33.6% of card length for top 100 cards, while Considerations for Using the Data makes up 2.1%.
- Content Comprehensiveness had the strongest correlation with human-rated overall quality (coefficient 0.3935, p=3.67E-07); removing Usage sections lowered BERT-predicted download score by 0.0185.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove the Usage section | Hugging Face dataset cards with a Usage section · Dataset cards stratified by creation month | predicted download score: 0.3917 -> 0.3732; -0.0185 | Hurt |
| Include a Usage section | Hugging Face dataset documentation · Dataset cards stratified by creation month | predicted download score: 0.3732 -> 0.3917; +0.0185 | Helped |
| Provide more comprehensive dataset content | Dataset cards assessed by human annotators · Human evaluation of dataset card quality | overall quality correlation coefficient: Coefficient: 0.3935, p-value: 3.67E-07 | Helped |
Surprising findings
- Usage sections appear in 33.2% of cards despite not being in the community template.
- Dataset Creation and Considerations for Using the Data sections are consistently under-filled regardless of popularity.
Limitations
- Relies on a single platform (Hugging Face) with NLP-oriented concentration; downloads as popularity proxy may be incomplete.
- Content analysis uses word counts and topic modeling that may not capture documentation nuance.
- Human evaluation is preliminary with 150 cards and five annotators.
Experimental setup
Datasets: Hugging Face dataset cards (7,433 non-empty out of 24,065 repositories) Papers With Code GitHub repository data
Metrics: card completion rate section word-count proportion ICC correlation coefficient with human quality ratings predicted download score
Builds on: Datasheets for Datasets Data Nutrition Labels BERT
Closest papers by meaning
- The State of Data Curation at NeurIPS: An Assessment of Dataset Development Practices in the Datasets and Benchmarks Track (NeurIPS 2024 · similarity 0.86)
- Bridging the Data Provenance Gap Across Text, Speech, and Video (ICLR 2025 · similarity 0.85)
- A Systematic Review of NeurIPS Dataset Management Practices (NeurIPS 2024 · similarity 0.84)
- Croissant: A Metadata Format for ML-Ready Datasets (NeurIPS 2024 · similarity 0.84)
- CroissantMiner: Automated Extraction and Validation of Croissant Metadata for ML Datasets (NeurIPS 2026 · similarity 0.83)
- Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.