Research map / Large language models / AI and society
Improving Reproducibility in Machine Learning Research(A Report from the NeurIPS 2019 Reproducibility Program)
Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d'Alche-Buc, Emily Fox, Hugo Larochelle
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| expect (not require) code submission for accepted papers, versus earlier supplementary-only policy | NeurIPS conference policy · Code at camera-ready | % papers with code: <50% -> 74.4% | Improved |
| voluntary code policy on submission volume | NeurIPS 2019 · Submissions | papers submitted: 4856 -> 6743 | No clear effect |
| code available at submission (per checklist) | NeurIPS 2019 reviews · Reviewer score | reviewer score: positively associated, p<1e-08 | Improved |
| authors from industry versus academia under code policy | NeurIPS 2019 acceptance · Acceptance rate | acceptance rate | No clear effect |
| reviewers having code available during review | NeurIPS 2019 reviewers · Reviewer survey | reviewers finding code useful: 5298 provided, 2255 looked, 1315 useful | Mixed |
| reviewers whose submissions lacked code | NeurIPS 2019 reviewers · Reviewer survey | wished code available: 3881 | Improved |
| reviewers finding checklist useful | NeurIPS 2019 reviews · Reviewer score and acceptance | paper score, acceptance: 34% found useful; higher scores and acceptance | Improved |
| checklist answer yes versus no | NeurIPS 2019 acceptance · Acceptance rate per question | acceptance rate | Improved |
| checklist NA answers on figures and tables | NeurIPS 2019 acceptance · Acceptance rate | acceptance rate | No clear effect |
| shift from ICLR 2019 to NeurIPS 2019 challenge | Reproducibility Challenge · Participation | papers claimed: 90 -> 173 | Improved |
| challenge year ICLR 2019 to NeurIPS 2019 | Reproducibility Challenge · Participation | participating institutions: 35 -> 73 | Improved |
| ICML 2019 soft encouragement versus NeurIPS 2019 expectation | code at camera-ready · Code submission | % papers with code: 67% -> 74.4% | Improved |
Closest papers by meaning
- Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods (AAAI 2024 · similarity 0.88)
- Position: Let’s Strengthen Verifiability if We Can’t Enforce Reproducibility (NeurIPS 2026 · similarity 0.86)
- What Do Machine Learning Researchers Mean by “Reproducible”? (AAAI 2025 · similarity 0.85)
- Standardization of Post-Publication Code Verification is Possible with the Support of the Community (NeurIPS 2026 · similarity 0.83)
- Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track (NeurIPS 2025 · similarity 0.82)
- The Unreasonable Effectiveness of Open Science in AI: A Replication Study (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
