Research map / Large language models / AI and society
Peer review should constrain evaluative authority
Hanrui Wang, Timo Spinde, Isabella Habereder, Chun-Shien Lu, Isao Echizen
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that peer-review reform should constrain evaluative authority under persistent uncertainty rather than only improving evaluators or aggregating judgments.
Problem
Peer review under growing submission pressure must assure research quality while allocating scarce opportunities (e.g., high-impact venue publication), and reform efforts have focused on reducing uncertainty through evaluator improvement or judgment aggregation.
Approach
Shift the target of peer-review reform from reducing uncertainty via better evaluators or aggregation to explicitly constraining the authority that evaluative judgments carry when uncertainty persists.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| have human supervisors rate from one shared detailed AI critique instead of writing independent reviews | Dual-track human-AI mutual supervision (Track A) · Single ICLR'25 submission, 4 supervisors | rating spread: 3/6/6/8 -> 5/5/5/6 | Mixed |
| add AI scrutiny of human critiques | Dual-track human-AI mutual supervision (Track B) · Single ICLR'25 submission | counter-signals flagged | Improved |
| let the same AI generate and scrutinize its own critique (ablation) | Dual-track design (outside proposed design) · Single ICLR'25 submission | AI self-rating and flags: rating 8, no failure modes detected | Hurt |
| replace full human review with supervision of an AI critique | Reviewer workload · Questionnaire, 4 supervisors | perceived task difficulty: 4 of 4 easier | Improved |
| add dual-track supervision for the area chair | Area chair adjudication · Single ICLR'25 submission | adjudication difficulty | No clear effect |
Closest papers by meaning
- Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards (ICML 2025 · similarity 0.87)
- Paper Copilot: Tracking the Evolution of Peer Review in AI Conferences (ICLR 2026 · similarity 0.84)
- From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review (NeurIPS 2025 · similarity 0.84)
- Eliciting Honest Information from Authors Using Sequential Review (AAAI 2024 · similarity 0.83)
- Dissecting Submission Limit in Desk-Rejections: A Mathematical Analysis of Fairness in AI Conference Policies (ICML 2025 · similarity 0.82)
- Position: The Artificial Intelligence and Machine Learning Community Should Adopt a More Transparent and Regulated Peer Review Process (ICML 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
