Research map / Large language models / AI and society
A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Recruit novice reviewers through auxiliary-review pretesting and mentor them during conference reviewing | ICML reviewer pool · ICML 2020 | meta-reviewer review quality: 2.08 -> 2.26 | Helped |
| Recruit novice reviewers through auxiliary-review pretesting and mentor them during conference reviewing | ICML reviewer pool · ICML 2020 | review length (symbols): 2858 -> 4759 | Helped |
| Recruit novice reviewers through auxiliary-review pretesting and mentor them during conference reviewing | ICML reviewer pool · ICML 2020 | reviewers submitting at least one review in time: 0.81 -> 0.92 | Helped |
| Recruit novice reviewers through auxiliary-review pretesting and mentor them during conference reviewing | ICML reviewer pool · ICML 2020 | positive bids per reviewer: 27.4 -> 34.6 | Helped |
| Recruit novice reviewers through auxiliary-review pretesting and mentor them during conference reviewing | ICML reviewer pool · ICML 2020 | discussion activity rate: 0.58 -> 0.68 | Helped |
| Recruit novice reviewers through auxiliary-review pretesting and mentor them during conference reviewing | ICML reviewer pool · ICML 2020 | post-rebuttal review update rate: 0.43 -> 0.61 | Helped |
| Recruit novice reviewers through auxiliary-review pretesting and mentor them during conference reviewing | ICML reviewer pool · ICML 2020 | initial overall score: 3.25 -> 3.34 | No clear effect |
| Recruit novice reviewers through auxiliary-review pretesting and mentor them during conference reviewing | ICML reviewer pool · ICML 2020 | self-reported confidence: 3.03 -> 3.05 | No clear effect |
| Recruit novice reviewers through auxiliary-review pretesting and mentor them during conference reviewing | ICML reviewer pool · ICML 2020 | self-reported expertise: 2.98 -> 2.83 | Hurt |
Closest papers by meaning
- Position: The Artificial Intelligence and Machine Learning Community Should Adopt a More Transparent and Regulated Peer Review Process (ICML 2025 · similarity 0.89)
- Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards (ICML 2025 · similarity 0.89)
- Dissecting Submission Limit in Desk-Rejections: A Mathematical Analysis of Fairness in AI Conference Policies (ICML 2025 · similarity 0.86)
- Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews (ICML 2024 · similarity 0.84)
- Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track (NeurIPS 2025 · similarity 0.84)
- CycleResearcher: Improving Automated Research via Automated Review (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.