Research map / Large language models / Language, safety and interpretability

Allocate Marginal Reviews to Borderline Papers Using LLM Comparative Ranking

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Uncertainty quantification for LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position paper proposing that conferences allocate marginal (surplus) reviews to papers in an LLM-identified borderline band, using pairwise LLM comparisons with Bradley-Terry ranking, with retrospective ICLR 2025 estimates of overlap and marginal review value.

Problem

Surplus reviewer capacity beyond the minimum review count is typically spread by load-balancing or affinity heuristics, and identifying borderline papers requires human reading that arrives too late to inform reviewer-count decisions at assignment time.

Approach

Use LLM pairwise comparisons aggregated with a Bradley-Terry model, before any human review, to define a borderline band that receives one extra review; LLM outputs are not used for accept/reject and are hidden from decision-makers.

Key results

• On 1000 ICLR 2025 papers with 40 rounds of GPT-5-mini pairwise comparisons, rho = 0.41 vs random baseline; Delta = 0.024 (95% CI [0.003, 0.045]).
• Expected net improved decisions about 24 for N=30,000, s=0.3 under the paper's first-order formula.
• In a 20-paper run, GPT-5 reached rho approx 0.8 at marginal fraction 0.5, versus about 0.5 for DeepSeek-V3-0324.

Surprising findings

• Overlap with human borderline set is higher for higher-capability judges, with DeepSeek-V3 near the random baseline trend.
• Authors report LLM ranking is better at identifying top papers than near-cutoff papers, as overlap increases at higher acceptance percentiles.

Limitations

• Retrospective proxies: human ordering built from decision tiers and mean scores, not ground-truth quality; Delta is from a calibrated logistic flip model, not causal.
• Small sample (1000 papers) and small ablation runs (20 and 100 papers) with limited statistical power.
• Possible LLM bias across paper categories and gaming concerns not empirically tested.

Experimental setup

Datasets: ICLR 2025 submissions with public reviews and decisions (1000-paper sample)

Benchmarks: ICLR 2025 retrospective sample (1000 papers)

Metrics: borderline overlap fraction (rho) marginal review flip-rate difference (Delta) Mann-Whitney AUC expected net improved decisions

Baselines: random allocation Toronto Paper Matching System

Builds on: Bradley-Terry model Large Conference Matching

Closest papers by meaning

• From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review (NeurIPS 2025 · similarity 0.84)
• PiCO: Peer Review in LLMs based on Consistency Optimization (ICLR 2025 · similarity 0.82)
• Are Key-Phrases All That Reviewers Care About? A Comprehensive Benchmarking of Reviewer Matchmaking Systems (AAAI 2025 · similarity 0.82)
• Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.82)
• From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation (NeurIPS 2026 · similarity 0.81)
• PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies (ICLR 2026 · similarity 0.81)