Research map / Large language models / AI and society
Position: Let’s Strengthen Verifiability if We Can’t Enforce Reproducibility
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper that quantifies declining code availability at CVPR, ICCV, ICLR, ICML and NeurIPS (2021–2025) and proposes a reviewer-checked verification workflow (experiment logs and metric computation) as a lightweight alternative to enforced reproducibility.
Problem
Empirical results in ML papers are hard to reproduce and code is often unavailable, and there is no mechanism to verify reported experiments before publication.
Approach
Shift from reproducibility to verifiability: require authors to supply training/inference logs and metric-evaluation material (results files plus evaluation code) at submission, have reviewers check them with formal consistency ratings, and publish statuses alongside accepted papers, without requiring full code release.
Key results
- Share of accepted papers with train or test code declined in 2025 across all five conferences, with the decline dating to 2023 for CVPR and ICLR.
- Papers with official public code receive more than twice as many citations on average as those without code.
- A small user study with five experienced reviewers measured about 5 minutes for the short metric script and 45 to 60 minutes for the longer one.
Surprising findings
- Code sharing has recently decreased even as empirical-topic papers grow in proportion.
- Reportedly, about 20% of the most acknowledged repositories account for 81.5% of acknowledgements.
Limitations
- Verification does not certify full reproducibility, data-pipeline correctness, or output authenticity; result files may be forged.
- Code-availability statistics rely on an LLM pipeline with residual errors and no formal error bars; citation correlation is not causal.
- Workload estimate comes from a small five-reviewer study.
Experimental setup
Datasets: accepted papers of CVPR, ICCV, ICLR, ICML, NeurIPS 2021-2025 (about 55,377 papers) Semantic Scholar citation counts
Metrics: proportion of accepted papers with train or test code proportion with model weights mean and median citation counts by code availability reviewer workload time
Baselines: Paper2Code RRPR badge (ICPR) JAIR reproducibility badges NeurIPS reproducibility checklist
Builds on: Paper2Code ACM artifact badging terminology OpenReview
Closest papers by meaning
- Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods (AAAI 2024 · similarity 0.86)
- What Do Machine Learning Researchers Mean by “Reproducible”? (AAAI 2025 · similarity 0.85)
- Standardization of Post-Publication Code Verification is Possible with the Support of the Community (NeurIPS 2026 · similarity 0.84)
- The Unreasonable Effectiveness of Open Science in AI: A Replication Study (AAAI 2025 · similarity 0.83)
- Efficiently Verifiable Proofs of Data Attribution (NeurIPS 2025 · similarity 0.82)
- Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.