Research map / Large language models / AI and society
Position: The Artificial Intelligence and Machine Learning Community Should Adopt a More Transparent and Regulated Peer Review Process
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
A position paper arguing AI/ML conferences should adopt a more transparent, open, and regulated peer review process, supported by Paper Copilot traffic statistics, review-disclosure analysis, and a small interest survey.
Problem
Peer review at large AI/ML conferences varies in how openly reviews are disclosed, and there is little quantitative evidence of what transparency level the community actually wants or how it affects engagement and review quality.
Approach
Uses web analytics and review-score/reply statistics aggregated by Paper Copilot across fully open, partially open, and closed venues to argue that transparent, regulated open review better serves the community, while acknowledging counterarguments on plagiarism and IP.
Key results
- ICLR (fully open) attracted 414,096 views and 88,220 active users, roughly four times the views and six times the active users of NeurIPS (partially open).
- Google CTR stayed consistently high across venues, ranging from 66.08% to 86.49%, regardless of disclosure mode.
- In a survey of 228 respondents, 57% said they would share review scores anonymously at fully closed venues such as CVPR 2025.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use fully open rather than fully closed review | AI/ML conference review processes · Conference statistics and community engagement | community interaction: ICLR ... 414,096 views, 88,220 active users ... nearly four times more views and six times more active users than NeurIP | Helped |
| Use fully open rather than partially open review | AI/ML conference review processes · ICLR 2024 vs NeurIPS 2024 accepted-paper discussions | maximum discussion replies: 76 compared to 49 | Helped |
| Use fully open rather than partially open review | AI/ML conference review processes · Accepted papers, ICLR and NeurIPS 2024 | high-confidence ratings | Hurt |
| Adopt open, partially open, or closed review rather than changing between these disclosure modes | AI/ML conference review processes · Review confidence, 2022–2024 | review confidence: averaging between 3.5 and 3.6 | No clear effect |
| Make review scores available for anonymous sharing | Fully closed-review conferences such as CVPR · CVPR 2025 survey | respondents willing to share scores: 57% | Helped |
Surprising findings
- Reviewer confidence distributions for accepted papers were similar across fully open ICLR and partially open NeurIPS, and the actual transparency level of venues changed little over the decade despite platform migrations.
Limitations
- Engagement metrics (views, sessions) are confounded by the site's own audience and search traffic, so they do not isolate the effect of review openness.
- Community submissions and survey responses are voluntary and may not represent the field's diversity.
- Several central claims, such as reviewer struggles and name-change inconsistencies, are anecdotal and not quantified.
Experimental setup
Datasets: ICLR review data via API NeurIPS review data via API ICML and CVPR community-submitted review data Paper Copilot community Google Form submissions (3,876 responses) Interest survey (228 responses)
Metrics: page views active users average engagement time click-through rate review confidence mean and standard deviation reply counts survey response rate
Builds on: OpenReview
Closest papers by meaning
- Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards (ICML 2025 · similarity 0.87)
- Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews (ICML 2024 · similarity 0.85)
- CycleResearcher: Improving Automated Research via Automated Review (ICLR 2025 · similarity 0.85)
- Position: AI/ML Influencers Have a Place in the Academic Process (ICML 2024 · similarity 0.84)
- Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track (NeurIPS 2025 · similarity 0.83)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.