Research map / Large language models / AI and society
CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers
Hexuan Deng, Xiaopeng Ke, Yichen Li, Ruina Hu, Dehao Huang, Derek F. Wong, Yue Wang, Xuebo Liu, Min Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use overlap metrics (BLEU, ROUGE-L, BERTScore) instead of LLM-judge category metrics | AI reviewer evaluation · CoCoReviewBench | ranking of reviewers | Hurt |
| aggregate all reviewers per paper instead of one reviewer | human reference coverage · 3,900 ICLR/NeurIPS papers | subcategories covered (of 23): 5.10 -> 9.23 | Improved |
| use reasoning models instead of non-reasoning models as AI reviewers | AI reviewers · CoCoReviewBench grounding and verifiability | grounding / verifiability | Improved |
| use strong closed-source models as AI reviewers instead of other models | AI reviewers · CoCoReviewBench | correctness and thoroughness vs human: GPT-5.2 +0.36 correctness, +0.64 thoroughness | Improved |
| fine-tune specialized AI reviewers on human reviews | CycleReviewer, OpenReviewer, SEA-E · CoCoReviewBench | correctness vs human: -0.08 to -0.53 | No clear effect |
| AI reviews versus human reviews on clarity | AI reviewers · CoCoReviewBench | clarity: Average +0.20 | Improved |
| AI reviews versus human reviews on category completeness | AI reviewers · CoCoReviewBench | category completeness (%): 55.66 -> 70.31 (average) | Improved |
| AI reviews versus human reviews on thoroughness within category for non-thinking models | non-reasoning AI reviewers · CoCoReviewBench | thoroughness: Llama-3.3-70B -0.23 | Hurt |
| treat incorrect opinions as references | correctness evaluation · CoCoReviewBench | AI vs human correctness gap | Improved |
| stronger conflict (>=5 incorrect opinions) in meta-review adjudication | meta-reviews · Meta-review quality | average score: 2.94 -> 3.24 | Improved |
| train Qwen3-8B on benchmark data for review classification | Qwen3-8B · ReviewClassify | entirely correct classifications: 87.09% of reviews correct | Improved |
Closest papers by meaning
- Fantastic Bugs and Where to Find Them in AI Benchmarks (NeurIPS 2025 · similarity 0.87)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.85)
- BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices (NeurIPS 2024 · similarity 0.85)
- Precise but Uncoupled: Reviewer Precision Does Not Guarantee Critique Uptake in Multi-Agent Math Reasoning (NeurIPS 2026 · similarity 0.84)
- Distinguishing Performance From Competence in Evaluations of Humanlike Abstract Reasoning (NeurIPS 2026 · similarity 0.84)
- Inverse Constitutional AI: Compressing Preferences into Principles (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
