Research map / Multi-agent systems and games / LLM agents
RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RE-Bench, 7 open-ended ML research engineering environments with 44 human expert runs and 35 Claude-3.5-Sonnet agent runs, showing agents beat humans early but plateau while humans keep improving.
Problem
Frontier AI safety policies flag autonomous AI R&D as a key risk, but there are few realistic AI R&D evaluations with direct human-expert comparison.
Approach
Continuous-score ML research engineering environments with normalized scores and matched 8-hour human baselines, designed around five desiderata (low floor, high ceiling, few issues, novelty, coverage) and compared directly against agents on the same resources.
Key results
- Claude-3.5-Sonnet agents outpace humans in the first hour but plateau; humans make steady progress over 8 hours (44 human attempts vs 35 agent 2-hour runs).
- Within 2 hours, agent best-of solutions reached normalized score around 0.52 on GPT-2 RL, above what most human experts achieved in 2 hours.
- Of 44 human attempts, 37 scored above 0.05 by 8 hours versus 15 by 2 hours.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Agents given 2h versus human experts given 8h | Claude-3.5-Sonnet agent vs human experts · RE-Bench 7 environments | normalized score over time | Mixed |
| Claude-3.5-Sonnet agent scaffold (no explicit elicitation) | Agents on 7 RE-Bench environments · RE-Bench | meaningful progress environments | Mixed |
| Standard tricks (e.g. PyTorch jit) used by agent instead of triton kernel | Optimize a kernel environment · RE-Bench | normalized score | Helped |
| Adding a 2-hour time-limit caveat: agents not given 8 hours | Claude-3.5-Sonnet agent · RE-Bench | plateau timing | No clear effect |
Surprising findings
- Claude-3.5-Sonnet found strong near-zero-shot solutions in Optimize kernel (using torch.jit) and GPT-2 RL, beating the worst 8-hour human submissions.
- Agents showed inconsistent rule understanding, e.g., violating the no-hyperparameter-change rule or the no-division rule without detection by the scorer.
Limitations
- Only 8 hours of work and 7 environments, making the evaluation noisy.
- Agents were not evaluated with proper elicitation (no better tools, parallel runs, or longer horizons).
Experimental setup
Datasets: OpenWebText Stanford Alpaca CodeContests
Benchmarks: RE-Bench
Metrics: normalized score percentage of runs above 0.05 progress success rate
Baselines: Claude-3.5-Sonnet agent human ML experts
Closest papers by meaning
- InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.89)
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.88)
- \$OneMillion-Bench: How Far are Language Agents from Human Experts? (NeurIPS 2026 · similarity 0.88)
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (ICLR 2025 · similarity 0.88)
- MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges? (NeurIPS 2025 · similarity 0.88)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.