AI Paper Atlas

Research map / Large language models / AI and society

CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers

Hexuan Deng, Xiaopeng Ke, Yichen Li, Ruina Hu, Dehao Huang, Derek F. Wong, Yue Wang, Xuebo Liu, Min Zhang

ICML 2026 · Large language models › AI and society · approach: Data commons and human baselines · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use overlap metrics (BLEU, ROUGE-L, BERTScore) instead of LLM-judge category metricsAI reviewer evaluation · CoCoReviewBenchranking of reviewersHurt
aggregate all reviewers per paper instead of one reviewerhuman reference coverage · 3,900 ICLR/NeurIPS paperssubcategories covered (of 23): 5.10 -> 9.23Improved
use reasoning models instead of non-reasoning models as AI reviewersAI reviewers · CoCoReviewBench grounding and verifiabilitygrounding / verifiabilityImproved
use strong closed-source models as AI reviewers instead of other modelsAI reviewers · CoCoReviewBenchcorrectness and thoroughness vs human: GPT-5.2 +0.36 correctness, +0.64 thoroughnessImproved
fine-tune specialized AI reviewers on human reviewsCycleReviewer, OpenReviewer, SEA-E · CoCoReviewBenchcorrectness vs human: -0.08 to -0.53No clear effect
AI reviews versus human reviews on clarityAI reviewers · CoCoReviewBenchclarity: Average +0.20Improved
AI reviews versus human reviews on category completenessAI reviewers · CoCoReviewBenchcategory completeness (%): 55.66 -> 70.31 (average)Improved
AI reviews versus human reviews on thoroughness within category for non-thinking modelsnon-reasoning AI reviewers · CoCoReviewBenchthoroughness: Llama-3.3-70B -0.23Hurt
treat incorrect opinions as referencescorrectness evaluation · CoCoReviewBenchAI vs human correctness gapImproved
stronger conflict (>=5 incorrect opinions) in meta-review adjudicationmeta-reviews · Meta-review qualityaverage score: 2.94 -> 3.24Improved
train Qwen3-8B on benchmark data for review classificationQwen3-8B · ReviewClassifyentirely correct classifications: 87.09% of reviews correctImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy