AI Paper Atlas

Research map / Large language models / AI and society

Design and Analysis of the NIPS 2016 Review Process

Nihar B. Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, Ulrike von Luxburg

JMLR 2018 · Large language models › AI and society · approach: Benchmarking and reproducibility practice · 64 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add volunteer author-reviewers (pool 2) to invited reviewersNIPS 2016 reviewer pool · NIPS 2016 mean scoresmean score: only clarity differs significantly; quality p=0.0061, clarity p=1.91 x 10^{-04}No clear effect
compare volunteer vs invited reviewers on inter-reviewer disagreementNIPS 2016 reviewer pool · NIPS 2016 consistencyfraction of disagreements: novelty p=0.9269, quality p=0.8648, impact p=0.7296, clarity p=0.8029No clear effect
reviewer seniority (senior vs PhD student)NIPS 2016 reviewers · NIPS 2016self-reported confidence: effect size 0.3577, p=3.308 x 10^{-57}Improved
add author rebuttal and discussion roundNIPS 2016 review process · NIPS 2016reviewer scores: 1193 of 12154 reviews changed; change in mean score less than 0.1No clear effect
invited vs volunteer reviewers participating in discussionNIPS 2016 discussion · NIPS 2016posts per reviewer-paper pair: p = 1.36 x 10^{-4}, effect size 0.0704Improved
non-student vs student reviewers participating in discussionNIPS 2016 discussion · NIPS 2016posts per reviewer-paper pair: p = 3.016 x 10^{-4}, effect size 0.0695Improved
2016 review process compared with 2015NIPS review process · NIPS messy middlesize of messy middle: 45% -> 30%Improved
use ordinal rankings instead of cardinal scoresNIPS 2016 review data · NIPS 2016ties: more than 30% ties in cardinal scores; none in ordinalImproved
restrict analysis to top 2k papersordinal ranking vs final decision · NIPS 2016disagreement with final decisions: 16-17% -> 27-28%Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy