Research map / Large language models / AI and society

From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review

NeurIPS 2025 · Large language models › AI and society · approach: Critiques of LLM claims and judging

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes replacing LLM paper-level rating with LLM agents doing pairwise paper comparisons aggregated by Bradley-Terry, and shows it better identifies high-citation papers than GPT rating baselines while introducing novelty and institutional biases.

Problem

Existing LLM-based peer review replicates traditional workflows with per-paper scoring, which is poorly calibrated (e.g., ratings concentrate around 6-7); the paper asks whether pairwise LLM comparisons aggregated into rankings can better identify high-impact papers.

Approach

LLM agents each judge one random pair of submissions with a binary preference; outcomes are aggregated with a Bradley-Terry maximum-likelihood model to produce a global ranking and accept/reject decisions, instead of absolute ratings and meta-review.

Key results

• At maximum scale, the GPT ranking system without area control achieves 20.00 average citations vs 19.36 for human acceptance, 8.56 for random, and 11.41 for the GPT rating system.
• 41.0% of human-accepted ICLR 2024 papers were also accepted by the GPT ranking system (42.2% for ICLR 2023).
• Papers selected by the GPT ranking system show smaller nearest-neighbor distances (lower novelty) and higher Gini coefficients (e.g., 43.8% of top-tier ICLR 2023 papers from 10 institutions vs 27.0% for humans).

Measured effects reported in the paper

Surprising findings

• GPT rating outputs were concentrated around 6 and 7, limiting differentiation and yielding only minor gains over random.
• The pairwise GPT ranking system matched or approached human-accepted citation averages despite human papers having visibility advantages from conference presentation.

Limitations

• Citation counts are a proxy for impact and are confounded by visibility and presentation advantages of human-accepted papers.
• Results rely only on GPT-4o mini for main experiments; other LLMs were tested only in the appendix.
• Strong institutional concentration and reduced topic novelty among selected papers.

Experimental setup

Datasets: ICLR 2023 submissions ICLR 2024 submissions NeurIPS 2023 submissions EMNLP 2023 submissions CoRL 2023 submissions

Benchmarks: ICLR 2023 ICLR 2024 EMNLP 2023 CoRL 2023 NeurIPS 2023

Metrics: average citation count of accepted papers citation count differences across decision tiers decision agreement with human decisions nearest-neighbor embedding distance (novelty) Gini coefficient of institutional concentration Spearman correlation with human review scores

Baselines: human acceptance decisions random acceptance GPT rating system Claude ranking system Gemini ranking system

Builds on: Bradley-Terry model AI Scientist-style LLM review pipelines CycleReviewer DeepReview

Closest papers by meaning

• PiCO: Peer Review in LLMs based on Consistency Optimization (ICLR 2025 · similarity 0.89)
• PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies (ICLR 2026 · similarity 0.87)
• Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews (ICML 2024 · similarity 0.86)
• CycleResearcher: Improving Automated Research via Automated Review (ICLR 2025 · similarity 0.86)
• Are Key-Phrases All That Reviewers Care About? A Comprehensive Benchmarking of Reviewer Matchmaking Systems (AAAI 2025 · similarity 0.86)
• RevisEval: Improving LLM-as-a-Judge via Response-Adapted References (ICLR 2025 · similarity 0.86)