Research map / Learning theory and algorithms / Prediction and decision losses
A Step Toward Quantifying Independently Reproducible Machine Learning Research
Edward Raff
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| higher paper readability (fewer reads to implement) | paper property · Reproducibility of 255 ML papers | reproduction success: p=9.68e-25 | Improved |
| empirical or balanced rather than theory-oriented paper | paper property · Reproducibility of 255 ML papers | reproduction success: p=1.55e-9 | Improved |
| authors replying to questions | paper property · Reproducibility of 255 ML papers | reproduction success: 22/26 vs 1/24 reproduced | Improved |
| hyper-parameters specified | paper property · Reproducibility of 255 ML papers | reproduction success: p=8.45e-6 | Improved |
| more tables per page | paper property · Reproducibility of 255 ML papers | reproduction success: p=0.010 | Improved |
| more equations per page | paper property · Reproducibility of 255 ML papers | reproduction success: p=0.004 | Hurt |
| higher algorithm difficulty | paper property · Reproducibility of 255 ML papers | reproduction success: p=2.94e-5 | Hurt |
| needing cluster compute | paper property · Reproducibility of 255 ML papers | reproduction success: p=8.75e-5 | Hurt |
| including step-code pseudo code | paper property · Reproducibility of 255 ML papers | reproduction success: p=2.31e-4 | Mixed |
| authors releasing code | paper property · Reproducibility of 255 ML papers | reproduction success: p=0.213 | No clear effect |
| year published | paper property · Reproducibility of 255 ML papers | reproduction success: p=0.964 | No clear effect |
| including conceptualization figures | paper property · Reproducibility of 255 ML papers | reproduction success: p=0.365 | No clear effect |
Closest papers by meaning
- Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods (AAAI 2024 · similarity 0.88)
- Position: Let’s Strengthen Verifiability if We Can’t Enforce Reproducibility (NeurIPS 2026 · similarity 0.88)
- What Do Machine Learning Researchers Mean by “Reproducible”? (AAAI 2025 · similarity 0.87)
- Standardization of Post-Publication Code Verification is Possible with the Support of the Community (NeurIPS 2026 · similarity 0.84)
- The Unreasonable Effectiveness of Open Science in AI: A Replication Study (AAAI 2025 · similarity 0.83)
- The Emergence of Reproducibility and Consistency in Diffusion Models (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
