Research map / Multi-agent systems and games / LLM agents

EXP-Bench: Can AI Conduct AI Research Experiments?

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces EXP-Bench, a benchmark of 461 end-to-end AI research experiment tasks extracted semi-autonomously from 51 NeurIPS/ICLR 2024 papers and their code, and evaluates OpenHands and IterativeAgent with several LLMs, finding very low end-to-end success.

Problem

Current AI agents struggle to carry out rigorous, end-to-end AI research experiments (design, implementation, execution, conclusion), and existing benchmarks isolate sub-parts or rely on simplified settings.

Approach

A semi-automated pipeline that extracts research questions from papers via multi-pass LLM extraction, pairs them with validated script chains from the open-source code, and grades agents with a conjunctive multi-metric judge (design, implementation, execution, conclusion, monitor) rather than a single success measure.

Key results

• OpenHands with o3-mini reaches 20.3% implementation and 18.4% design correctness, but only 1.4% on the strict all-correct metric and 0.5% when execution is also required.
• OpenHands with Claude-3.7 Sonnet achieves 35.0% implementation and 14.9% combined implementation-execution scores, with 0.4% on All-with-execution.
• Conjunctive metrics drop average scores from 20.6% (monitor only) to 0.2% after adding design, conclusion, implementation and execution checks.

Surprising findings

• Agents often produced plausible responses without actually running experiments, inflating partial design and implementation scores.
• Little correlation was found between agent runtime or cost and overall performance.

Limitations

• Focuses only on the experimentation procedure, not literature review, ideation, or the full research lifecycle.
• Execution-based metric can overestimate correctness, since mock implementations may run successfully.
• Evaluates only a subset of traces with execution due to time cost.

Experimental setup

Datasets: NeurIPS 2024 papers and code ICLR 2024 papers and code

Benchmarks: EXP-Bench

Metrics: design correctness (D) implementation correctness (I) execution (E) conclusion correctness (C) monitor integrity (M) All-check All-check with execution

Baselines: OpenHands IterativeAgent

Builds on: OpenHands PaperBench MLE-Bench

Closest papers by meaning

• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.88)
• MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.88)
• InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.87)
• RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts (ICML 2025 · similarity 0.87)
• PaperBench: Evaluating AI’s Ability to Replicate AI Research (ICML 2025 · similarity 0.87)
• From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking (ICLR 2026 · similarity 0.87)