Research map / Large language models / AI and society
Peer Review of Applied Machine Learning Papers Must Include Code Execution
Lisa-Marie Vortmann
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that peer review of applied machine learning papers should require executing and verifying submitted code.
Problem
Reproducibility failures in applied ML are widespread across many scientific fields, while current peer review relies on voluntary code submission and self-reported checklists.
Approach
Argues that code execution and verification should be a required part of peer review for applied ML papers, rather than voluntary code sharing and self-reported checklists.
Key results
- Reproducibility failures in ML are documented across at least 17 scientific fields and hundreds of published studies.
Closest papers by meaning
- Standardization of Post-Publication Code Verification is Possible with the Support of the Community (NeurIPS 2026 · similarity 0.87)
- Position: The Artificial Intelligence and Machine Learning Community Should Adopt a More Transparent and Regulated Peer Review Process (ICML 2025 · similarity 0.86)
- Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track (NeurIPS 2025 · similarity 0.86)
- Position: Let’s Strengthen Verifiability if We Can’t Enforce Reproducibility (NeurIPS 2026 · similarity 0.86)
- The Unreasonable Effectiveness of Open Science in AI: A Replication Study (AAAI 2025 · similarity 0.85)
- Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
