Research map / Learning theory and algorithms / Prediction and decision losses
On Testing for Biases in Peer Review
Ivan Stelmakh, Nihar Shah, Aarti Singh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Tomkins et al. Wald test with noisy double-blind score estimates when true scores correlate with the author property | Tomkins et al. peer-review test · Synthetic simulations | Type-I error | Hurt |
| Use the Tomkins et al. Wald test when the reviewers’ true decision model differs from its assumed linear logistic form | Tomkins et al. peer-review test · Synthetic simulations | Type-I error | Hurt |
| Treat multiple decisions by the same reviewer as independent when reviewer calibration correlates them | Tomkins et al. peer-review test · Synthetic simulations | Type-I error | Hurt |
| Allow single-blind reviewers to see author identities while bidding | Tomkins et al. experimental setup · Synthetic simulations | Type-I error | Hurt |
| Use a popular similarity-based reviewer assignment procedure without the paper’s joint assignment design | Tomkins et al. experimental setup · Synthetic simulations | Type-I error | Hurt |
| Use the Disagreement test with the proposed experimental setup | Peer-review bias testing · Absolute bias problem | Type-I error control and non-trivial power | Improved |
| Use the Counting test with the proposed experimental setup | Peer-review bias testing · Absolute bias problem | Type-I error control and non-trivial power | Improved |
| Use the Disagreement test instead of the Counting test under the generalized logistic model | Peer-review bias testing · Generalized logistic model | Type-I error control and non-trivial power | Improved |
| Use the Counting test instead of the Disagreement test under the generalized linear model | Peer-review bias testing · Generalized linear model | Type-I error control and non-trivial power | Improved |
| Use the Disagreement test under the generalized linear model | Peer-review bias testing · Generalized linear model | Type-I error | Hurt |
| Use the Counting test under the generalized logistic model | Peer-review bias testing · Generalized logistic model | Type-I error | Hurt |
| Use the Disagreement test rather than the Counting test under the absolute bias problem | Peer-review bias testing · Absolute bias problem | Power | Improved |
Closest papers by meaning
- Uncovering Latent Biases in Text: Method and Application to Peer Review (AAAI 2021 · similarity 0.90)
- From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review (NeurIPS 2025 · similarity 0.84)
- Trust, but Don’t Verify: Epistemic Blind Spots in LLM Source Evaluation (NeurIPS 2026 · similarity 0.82)
- Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards (ICML 2025 · similarity 0.82)
- Position: Rethinking LLM Bias Probing Using Lessons from the Social Sciences (ICML 2025 · similarity 0.82)
- PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies (ICLR 2026 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
