Research map / Large language models / AI and society

Standardization of Post-Publication Code Verification is Possible with the Support of the Community

NeurIPS 2026 · Large language models › AI and society · approach: Benchmarking and reproducibility practice

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position paper arguing journals and proceedings should adopt a post-publication code verification system, extending ACM reproduced/replicated badges with up to two verification badges per article in metadata.

Problem

Post-publication verification of machine learning results is limited and unformalized; pre-publication review and artifact evaluation cannot ensure reported results are independently verifiable.

Approach

Extend ACM pre-publication Results Reproduced/Replicated badges to a journal-run post-publication process where independent teams submit verification reports, with up to one reproduction and one replication badge displayed in article metadata.

Key results

• Proposes a two-badge post-publication verification system (at most one Results Reproduced and one Results Replicated badge per article) linked to verified code repositories in article metadata.
• Surveys reproducibility policies of 11 journals, 8 ML conferences, and 3 ACM-affiliated venues, finding no post-publication verification mechanisms in the surveyed journals.

Surprising findings

• The paper reports that none of the surveyed journals currently has a post-publication verification mechanism.
• Papers with Code went offline in mid-2025, illustrating the risk of relying on third-party platforms for scientific infrastructure.

Limitations

• Added editorial workload for journals and reviewers.
• Confidentiality and restricted datasets (e.g., EHR) can prevent independent verification.
• Reputation-based system may be gamed by fabricated verification reports.

Experimental setup

Builds on: ACM artifact badging

Closest papers by meaning

• Position: Let’s Strengthen Verifiability if We Can’t Enforce Reproducibility (NeurIPS 2026 · similarity 0.84)
• Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track (NeurIPS 2025 · similarity 0.81)
• ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code (NeurIPS 2025 · similarity 0.81)
• The Unreasonable Effectiveness of Open Science in AI: A Replication Study (AAAI 2025 · similarity 0.81)
• Position: The Artificial Intelligence and Machine Learning Community Should Adopt a More Transparent and Regulated Peer Review Process (ICML 2025 · similarity 0.81)
• Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards (ICML 2025 · similarity 0.80)