Research map / Large language models / AI and society
LitReview Arena: Evaluating Literature Review Agents with Battle-style Peer Review Platform
Ruotong Zhao, zhiyu chen, Xurui Liu, Haidong Xue, Dong Liang, Jigao Fu, Wu YanBiao, Yuanyi Zhen, Fengli Xu, Yong Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use agentic deep research systems instead of base language models | literature review generation · LitReviewBench | Average overall utility Elo: 721.1 -> 1181.6 | Improved |
| use agentic systems (more test-time compute) | literature review generation · LitReviewBench | token cost per query: 8.098K -> 122.297K | Hurt |
| allocate more computation to search and synthesis | agentic systems · LitReviewBench vs human drafts | win rate vs humans (D5): 23.0% | No clear effect |
| increase draft length or interaction turns | literature review agents · LitReviewBench | D3/D4 scores | No clear effect |
| use an uncalibrated LLM judge on retrieval-like criterion (D1) | Qwen3-235B judge · LitReviewBench D1 | Spearman rho with experts: 0.552 | Mixed |
| use an uncalibrated LLM judge on synthesis criteria | Qwen3-235B judge · LitReviewBench D2-D4 | Spearman rho with experts: 0.442 (D2), 0.467 (D3), 0.430 (D4) | Hurt |
| calibrate judge with structure-, content-similar demonstrations and gap anchors (LitJudge) | Qwen3-235B judge · LitReviewBench D5 | alignment with experts: 0.467 -> 0.792 | Improved |
| add expert-derived gap anchors with MMR selection for D4 | LitJudge · LitReviewBench D4 | alignment with experts: 0.430 -> 0.842 | Improved |
| calibrate judge on Literature Coverage | LitJudge · LitReviewBench D1 | alignment with experts: 0.552 -> 0.576 | Improved |
| calibrate judge for Claim Support and Paper Structure | LitJudge · LitReviewBench D2, D3 | alignment with experts: 0.442 -> 0.673 (D2); 0.467 -> 0.649 (D3) | Improved |
| use LitJudge calibration versus random few-shot ICL control (ablation) | LitJudge · LitReviewBench | alignment with experts | Improved |
| use specialized systems Open Deep Research and SurveyForge | literature review generation · LitReviewBench | expert preference | Improved |
Closest papers by meaning
- CycleResearcher: Improving Automated Research via Automated Review (ICLR 2025 · similarity 0.88)
- Agent Reviewers: Domain-specific Multimodal Agents with Shared Memory for Paper Review (ICML 2025 · similarity 0.88)
- From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review (NeurIPS 2025 · similarity 0.87)
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (ICLR 2024 · similarity 0.87)
- SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks (NeurIPS 2025 · similarity 0.87)
- Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
