Research map / Large language models / AI and society
Design and Analysis of the NIPS 2016 Review Process
Nihar B. Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, Ulrike von Luxburg
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add volunteer author-reviewers (pool 2) to invited reviewers | NIPS 2016 reviewer pool · NIPS 2016 mean scores | mean score: only clarity differs significantly; quality p=0.0061, clarity p=1.91 x 10^{-04} | No clear effect |
| compare volunteer vs invited reviewers on inter-reviewer disagreement | NIPS 2016 reviewer pool · NIPS 2016 consistency | fraction of disagreements: novelty p=0.9269, quality p=0.8648, impact p=0.7296, clarity p=0.8029 | No clear effect |
| reviewer seniority (senior vs PhD student) | NIPS 2016 reviewers · NIPS 2016 | self-reported confidence: effect size 0.3577, p=3.308 x 10^{-57} | Improved |
| add author rebuttal and discussion round | NIPS 2016 review process · NIPS 2016 | reviewer scores: 1193 of 12154 reviews changed; change in mean score less than 0.1 | No clear effect |
| invited vs volunteer reviewers participating in discussion | NIPS 2016 discussion · NIPS 2016 | posts per reviewer-paper pair: p = 1.36 x 10^{-4}, effect size 0.0704 | Improved |
| non-student vs student reviewers participating in discussion | NIPS 2016 discussion · NIPS 2016 | posts per reviewer-paper pair: p = 3.016 x 10^{-4}, effect size 0.0695 | Improved |
| 2016 review process compared with 2015 | NIPS review process · NIPS messy middle | size of messy middle: 45% -> 30% | Improved |
| use ordinal rankings instead of cardinal scores | NIPS 2016 review data · NIPS 2016 | ties: more than 30% ties in cardinal scores; none in ordinal | Improved |
| restrict analysis to top 2k papers | ordinal ranking vs final decision · NIPS 2016 | disagreement with final decisions: 16-17% -> 27-28% | Hurt |
Closest papers by meaning
- Position: The Artificial Intelligence and Machine Learning Community Should Adopt a More Transparent and Regulated Peer Review Process (ICML 2025 · similarity 0.81)
- Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards (ICML 2025 · similarity 0.80)
- Allocate Marginal Reviews to Borderline Papers Using LLM Comparative Ranking (NeurIPS 2026 · similarity 0.80)
- Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track (NeurIPS 2025 · similarity 0.80)
- CycleResearcher: Improving Automated Research via Automated Review (ICLR 2025 · similarity 0.79)
- Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews (ICML 2024 · similarity 0.78)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
