Research map / Multi-agent systems and games / LLM agents

PaperBench: Evaluating AI’s Ability to Replicate AI Research

ICML 2025 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PaperBench, a benchmark of 20 ICML 2024 papers with hierarchical author-co-developed rubrics where agents must replicate results from scratch, graded by an LLM judge validated on JudgeEval.

Problem

Measuring whether AI agents can autonomously replicate state-of-the-art ML research end-to-end (understanding a paper, writing code from scratch, and running experiments) is not supported by existing benchmarks, which rely on existing codebases, Kaggle tasks, or lack fine-grained grading.

Approach

Rubric trees co-developed with original paper authors decompose each replication into weighted binary leaf requirements (Code Development, Execution, Result Match), graded by an LLM judge (SimpleJudge) and reproduced in a fresh VM; the judge itself is benchmarked on JudgeEval against human gold labels.

Key results

• Claude 3.5 Sonnet with BasicAgent achieves the highest average Replication Score at 21.0%, versus 13.2% for o1 and under 10% for other tested models.
• On a 3-paper subset, the best-of-3 human ML PhD baseline reaches 41.4% after 48 hours, compared with 26.6% for o1.
• o3-mini with SimpleJudge achieves F1 of 0.83 on JudgeEval at about $66 per paper, comparable to o1 (0.84) at roughly ten times the cost.

Measured effects reported in the paper

Surprising findings

• Models often end runs early, claiming completion or unsolvable problems; a forced-continuation IterativeAgent raised o1 from 13.2% to 24.4%, but lowered Claude 3.5 Sonnet's score from 21.0% to 16.1%.
• o1 initially outperforms human baseline but plateaus after the first hour and falls behind humans after about 24 hours.

Limitations

• Only 20 papers, and rubric creation is extremely labor-intensive (many tens of hours per paper).
• LLM judge is less accurate than expert humans and non-deterministic (best F1 0.83-0.84).
• Contamination risk from original authors' public codebases, which may inflate scores for future models.

Experimental setup

Datasets: 20 ICML 2024 Spotlight and Oral papers JudgeEval (human-graded partial replications)

Benchmarks: PaperBench PaperBench Code-Dev JudgeEval

Metrics: replication score F1 accuracy precision recall cost per paper

Baselines: o1 o3-mini GPT-4o DeepSeek-R1 Gemini 2.0 Flash Claude 3.5 Sonnet human ML PhD baseline

Builds on: ReAct Inspect AI basic agent RE-Bench CORE-Bench

Closest papers by meaning

• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.90)
• InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.89)
• MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.89)
• SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.88)
• BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices (NeurIPS 2024 · similarity 0.88)
• From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking (ICLR 2026 · similarity 0.88)