Research map / Learning theory and algorithms / Prediction and decision losses
A Principled Approach to Randomized Selection under Uncertainty: Applications to Peer Review and Grant Funding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MERIT, a polynomial-time maximin randomized top-k selection method over quality intervals that maximizes worst-case expected top-k count and enforces ex post validity, and compares it to Swiss NSF and randomize-above-threshold on peer review data.
Problem
Randomized selection of top-k candidates from error-prone evaluations is ad hoc; existing lotteries (Swiss NSF, randomize-above-threshold) lack principled justification and violate axioms like ex post validity, monotonicity in budget, and stability.
Approach
Model uncertainty as Knightian quality intervals with only their interval order, solve a maximin LP maximizing worst-case expected number of true top-k selected (via a polynomial-time separation oracle and cutting-plane method), then post-process marginals to enforce ex post validity and sample systematically.
Key results
- Under the Swiss NSF linear miscalibration model, MERIT matches Swiss NSF and model-adjusted deterministic selection in expected utility.
- Under the worst-case interval objective, MERIT achieves up to 0.19 higher utility than Swiss NSF.
- MERIT runs in under five minutes on a laptop for inputs with over 11,000 proposals.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MERIT instead of deterministic selection or Swiss NSF selection | Top-k selection under interval uncertainty · Worst-case over feasible rankings: Swiss NSF, NeurIPS 2024, and ICLR 2025 data | worst-case utility: up to 0.19 more than Swiss NSF | Helped |
| Use MERIT instead of Swiss NSF randomized selection or deterministic model-based selection | Top-k selection under the linear miscalibration model · Synthetic Swiss NSF and conference review data | expected utility | No clear effect |
| Select by raw mean score rather than accounting for reviewer miscalibration | Top-k selection under the linear miscalibration model · Synthetic review data | expected utility | Hurt |
| Increase the degree of reviewer miscalibration (ablation) | Selection methods under the Swiss NSF linear model · Synthetic Swiss NSF review data | expected utility | Mixed |
| Select the top one-tenth rather than the top one-third of proposals (ablation) | Selection methods under the Swiss NSF linear model · Synthetic review data | expected utility | No clear effect |
| Increase missing-review sparsity when generating Manski-bound intervals (ablation) | Selection methods on Swiss NSF grant-review data · Swiss NSF 2020, Manski bounds | worst-case utility | Mixed |
| Use MERIT instead of Swiss NSF or deterministic selection with Manski-bound intervals | Top-k selection on Swiss NSF grant-review data · Swiss NSF 2020, Manski bounds | worst-case utility | Helped |
Surprising findings
- Swiss NSF and MERIT both exhibit non-monotonicity in budget on real 2020 Swiss NSF review data (selection probability drops from 0.75 to 0.524 when k goes from 89 to 90).
- A Bayesian model implies a deterministic selection maximizes expected utility, so randomization is unnecessary under full Bayesian assumptions.
Limitations
- Assumes only interval ordering is used; suboptimal when well-calibrated probabilistic models exist.
- Assumes all proposals request the same budget; variable-cost extension is heuristic/future work.
- Less interpretable than single-threshold randomization methods.
Experimental setup
Datasets: Swiss NSF 2020 grant review data NeurIPS 2024 OpenReview reviews ICLR 2025 OpenReview reviews synthetic linear miscalibration review data
Benchmarks: Swiss NSF 2020 review data (simulated) NeurIPS 2024 review data (simulated) ICLR 2025 review data (simulated)
Metrics: expected fraction of top-k selected worst-case expected fraction of top-k selected runtime
Baselines: Swiss NSF selection algorithm randomize-above-threshold Deterministic (mean) Deterministic (model)
Builds on: Swiss NSF lottery method randomize-above-threshold Manski bounds ellipsoid method
Closest papers by meaning
- Smooth Partial Lotteries for Stable Randomized Selection (NeurIPS 2026 · similarity 0.88)
- Strategyproof Peer Selection: Mechanisms, Analyses, and Experiments (AAAI 2016 · similarity 0.86)
- Counterfactual Evaluation of Peer-Review Assignment Policies (NeurIPS 2023 · similarity 0.86)
- A One-Size-Fits-All Approach to Improving Randomness in Paper Assignment (NeurIPS 2023 · similarity 0.85)
- Learning Fair Policies for Multi-Stage Selection Problems from Observational Data (AAAI 2024 · similarity 0.83)
- Fair and Welfare-Efficient Constrained Multi-Matchings under Uncertainty (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.