Research map / Large language models / AI and society
Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
A position paper arguing AI conference peer review needs a two-stage bidirectional review system where authors rate reviews (with LLM-generated reviews shown as a reference) and a system-backed reviewer reward scheme of badges, impact scores, and in-kind rewards.
Problem
Declining review quality at major AI conferences amid submission surges (over 10,000 per venue), with reviewers facing no accountability and little reward, and growing LLM-generated reviews.
Approach
Proposes a two-stage release of reviews so authors rate review quality before seeing critical content and ratings, plus a system-issued reviewer badge and impact-score reward framework tied to author feedback scores.
Key results
- Review error rate reportedly fell from 7.79 to 5.54 per 1000 words (2017 to 2024), a 28.8% reduction, used as soft evidence of LLM use.
- Share of error-free ICLR reviews rose from 31.4% in 2023 to 37.8% in 2024.
- On average 17% of top-20 ICLR keywords change annually; 10 of 14 surveyed major AI conferences had a Reviewer Hall of Fame in 2024.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Review error frequency declined between 2017 and 2024 | ICLR peer reviews · ICLR 2017–2024 | spelling and grammar errors per 1000 words: 7.79 per 1000 words in 2017 to 5.54 in 2024; 28.8% reduction | Helped |
| Review error-free share increased from 2023 to 2024 | ICLR peer reviews · ICLR 2023–2024 | error-free reviews: 31.4% in 2023 to 37.8% in 2024 | Helped |
Surprising findings
- CVPR 2025 desk-rejected 19 papers authored by reviewers flagged as highly irresponsible by meta-reviewers.
- ICML 2024 initially planned in-kind compensation for the top 10% of reviewers but reduced the pool due to financial constraints.
Limitations
- Typo-based LLM-usage evidence is acknowledged as confounded by spell-checking tools and is not conclusive.
- Proposals are not empirically validated; the reviewer impact score and badge schemes are untested.
- Author feedback could be gamed by retaliatory or malicious authors, and reviewers could game badges via lenient reviews.
Experimental setup
Datasets: ICLR 2017-2024 OpenReview reviews ICLR 2018-2025 author-specified keywords
Metrics: typos per 1000 words proportion of error-free reviews top-20 keyword turnover
Builds on: NeurIPS 2021 review consistency experiment isotonic mechanism double-blind peer review
Closest papers by meaning
- Position: The Artificial Intelligence and Machine Learning Community Should Adopt a More Transparent and Regulated Peer Review Process (ICML 2025 · similarity 0.87)
- Dissecting Submission Limit in Desk-Rejections: A Mathematical Analysis of Fairness in AI Conference Policies (ICML 2025 · similarity 0.84)
- Eliciting Honest Information from Authors Using Sequential Review (AAAI 2024 · similarity 0.84)
- From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review (NeurIPS 2025 · similarity 0.84)
- Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews (ICML 2024 · similarity 0.83)
- Agent Reviewers: Domain-specific Multimodal Agents with Shared Memory for Paper Review (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.