Research map / Learning theory and algorithms / Prediction and decision losses
Eliciting Honest Information from Authors Using Sequential Review
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a sequential review mechanism that elicits truthful paper rankings from multi-submission authors by conditionally reviewing lower-ranked papers, assuming only that author utility is increasing in the quality of accepted papers.
Problem
Isotonic mechanism for eliciting author rankings of multiple submissions only works if author utility is increasing and convex in review score, which is often violated in peer review (e.g., authors maximizing accepted paper count).
Approach
Review an author's papers in the reported ranking order, conditioning review of each lower-ranked paper on the review outcomes of higher-ranked ones; misreporting causes earlier termination, so truthful ranking is incentivized under a monotonicity-based sufficient condition, with memoryless coin-flip and credit pool instances.
Key results
- Sequential review improves conference utility toward the isotonic upper bound by at least 40% across broad Gaussian settings when authors submit more than three papers.
- On ICLR 2021-2023 data, the threshold sequential review mechanism saves over 20% of review burden at equal conference utility.
- Even with two papers per author, review burden is reduced by 10-30% depending on the prior.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sequential review based on the author's ranking instead of unconditional parallel review | Conference review · Gaussian review setting | conference utility: at least a 40% improvement towards the upper bound | Helped |
| Use sequential review based on the author's ranking instead of unconditional parallel review | Conference review · ICLR OpenReview 2021–2023 | review burden: over 20% | Helped |
| Use sequential review based on the author's ranking instead of unconditional parallel review | Conference review · Gaussian review setting, n=2 | review burden: 10-30% | Helped |
| Increase the number of papers per author | Threshold sequential review · Gaussian review setting | conference utility relative to parallel review | Helped |
| Make papers more likely to be low quality | Threshold sequential review · Gaussian review setting | conference utility relative to parallel review | Helped |
| Increase review noise | Threshold sequential review · Gaussian review setting | conference utility relative to parallel review | Helped |
| Increase the number of papers per author | Threshold sequential review · Gaussian review setting | review burden | Helped |
| Use sequential review instead of parallel review | Authors choosing effort and paper quality · Binary effort setting | marginal rate of substitution for high- versus low-quality papers | Helped |
| Use sequential review instead of parallel review | Authors choosing effort and paper quality · Finite effort setting | marginal rate of substitution between higher- and lower-acceptance-probability p | Helped |
Surprising findings
- Without truthful ranking information, the isotonic mechanism can yield conference utility even worse than the parallel baseline.
Limitations
- Assumes each paper has a single author; coauthorship handled by assigning papers to one author.
- Sufficient (not necessary) truthfulness condition; theory assumes one reviewer per paper.
- Simulations assume authors know true quality or use simplified noise models; threshold mechanism only is evaluated empirically.
Experimental setup
Datasets: ICLR OpenReview 2021-2023
Benchmarks: ICLR OpenReview 2021-2023 (fitted softmax model)
Metrics: relative conference utility relative review burden average quality of reviewed papers
Baselines: parallel review mechanism isotonic mechanism with true ranking
Builds on: isotonic mechanism
Closest papers by meaning
- You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism (NeurIPS 2021 · similarity 0.88)
- Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards (ICML 2025 · similarity 0.84)
- A Principled Approach to Randomized Selection under Uncertainty: Applications to Peer Review and Grant Funding (NeurIPS 2025 · similarity 0.81)
- Allocate Marginal Reviews to Borderline Papers Using LLM Comparative Ranking (NeurIPS 2026 · similarity 0.81)
- From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review (NeurIPS 2025 · similarity 0.81)
- Dissecting Submission Limit in Desk-Rejections: A Mathematical Analysis of Fairness in AI Conference Policies (ICML 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.