Research map / Large language models / AI and society

Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track

NeurIPS 2025 · Large language models › AI and society · approach: Benchmarking and reproducibility practice · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position paper arguing ML conferences should create a "Refutations and Critiques" track to give peer-reviewed, official venues for correcting flawed published work, with an illustrative example critique of an ICLR 2025 Oral.

Problem

ML conferences lack official, reputable mechanisms to critique, correct, or retract flawed, misleading, or fraudulent papers once accepted, and informal channels (blogs, social media, OpenReview comments) lack rigor and visibility.

Approach

Proposes a dedicated Refutations and Critiques track inside major ML conferences, peer-reviewed with bespoke standards (correct/rigorous, substantive, constructive, significant), with original authors involved in review and critiques explicitly linked to the criticized paper, rather than reforming main-track peer review.

Key results

• Argues that conference peer review is fallible and post-hoc correction mechanisms are more practical than reforming review, supported mainly by anecdotes and examples such as the ICML 2022 Outstanding Paper privacy critique and the Distill critique of adversarial examples being cited about 2% as oft
• Proposes four review principles for R&C submissions: correct/rigorous, substantive, constructive, and significant.

Surprising findings

• A Distill critique of 'Adversarial Examples are not Bugs, They are Features' has only about 2% of the original's citations, illustrating low visibility of critiques outside main venues.

Limitations

• Relies largely on community anecdotes and personal experience because peer-reviewed evidence on post-publication critiques is scarce, as the authors acknowledge.
• No empirical evaluation of the proposed track; the illustrative ICLR 2025 example is a separate paper that is only referenced.
• Risks of adversarial or frivolous submissions and flawed refutations are discussed but not resolved.

Experimental setup

Builds on: NeurIPS Datasets and Benchmarks Track Machine Learning Reproducibility Challenge TMLR reproducibility certification

Closest papers by meaning

• Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods (AAAI 2024 · similarity 0.84)
• Position: The Artificial Intelligence and Machine Learning Community Should Adopt a More Transparent and Regulated Peer Review Process (ICML 2025 · similarity 0.83)
• PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies (ICLR 2026 · similarity 0.82)
• What Do Machine Learning Researchers Mean by “Reproducible”? (AAAI 2025 · similarity 0.82)
• Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews (ICML 2024 · similarity 0.82)
• Position: Let’s Strengthen Verifiability if We Can’t Enforce Reproducibility (NeurIPS 2026 · similarity 0.82)