Research map / Large language models / AI and society
The Unreasonable Effectiveness of Open Science in AI: A Replication Study
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
A systematic replication study of 30 highly cited AI papers (2012, 2014, 2016) that finds 50% of the 22 attempted papers were reproduced at least partly, with code+data sharing strongly tied to success and data documentation mattering more than code documentation.
Problem
The extent of the reproducibility crisis in AI research is unclear, and it is unknown which factors (code/data sharing, documentation quality) determine whether AI studies can be reproduced.
Approach
Attempts to reproduce highly cited empirical AI papers at the highest feasible reproducibility level (R3 data or R4 code+data), classifies results as success/partial/failure, and uses logistic regression and true positive rates over 20 problem types to identify what drives irreproducibility.
Key results
- Of 22 attempted studies (8 of 30 rejected), 6 were fully reproduced and 5 partially, a 50% success rate among those started.
- 86% (6 of 7) of R4 studies sharing code and data were reproduced fully or partly, versus 33% (5 of 15) of R3 studies sharing only data.
- Problems with data documentation (P15 dataset mismatch, P18 unspecified train/val/test partition) have TPR of 1.0 and strong logistic regression weights, while code documentation quality is not indicative of irreproducibility, except uninspectable compiled code (P6, TPR 1.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Share both code and data instead of data only | AI research studies · AI replication studies (R3 vs. R4) | inclusive success rate: 33% -> 86% | Helped |
| Document data work clearly, including dataset identity and train/validation/test partitioning | AI research studies · AI replication studies | reproducibility | Helped |
| Share code, even if code documentation is poor, partially missing, or not versioned | AI research studies · AI replication studies | reproducibility: 33% -> 86% | Helped |
| Share code in an uninspectable compiled form | AI research studies · AI replication studies | reproducibility | Hurt |
| Specify random seeds and random number generators | AI research studies · AI replication studies | reproducibility | Hurt |
Surprising findings
- Sharing code and data does not fully guarantee reproduction: 40 hours was insufficient even for many R4 studies.
- Quality of code documentation, partially missing code, or unversioned code did not correlate with irreproducibility as long as code was shared.
Limitations
- Small sample of 22 studies makes estimates highly uncertain.
- Sample of highly cited papers may overestimate reproducibility; 40-hour time limit may underestimate it.
- Implementation errors by the small reproduction team could affect results.
Experimental setup
Datasets: datasets of 30 highly cited AI papers from 2012, 2014, 2016 (original data as provided by authors or retrieved online)
Metrics: success rate (success / partial success / failure / no resul true positive rate per problem type logistic regression accuracy and feature weights conditional probabilities of reproducibility given sharing
Baselines: Hutson/Gundersen-style 63.5% reproducibility estimate (Gundersen and Kjensmo 201
Builds on: reproducibility type taxonomy (R1-R4) outcome reproducible vs analysis reproducible terminology
Closest papers by meaning
- PaperBench: Evaluating AI’s Ability to Replicate AI Research (ICML 2025 · similarity 0.85)
- Agent-Native Research Artifacts (NeurIPS 2026 · similarity 0.84)
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.84)
- Can AI Agents Synthesize Scientific Conclusions? (NeurIPS 2026 · similarity 0.84)
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.83)
- Position: Let’s Strengthen Verifiability if We Can’t Enforce Reproducibility (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.