Research map / Multi-agent systems and games / LLM agents

AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AstaBench, a 2400+-problem scientific research agent benchmark suite with a reproducible literature search environment, cost-aware leaderboard tooling, and a suite of 16 agent classes, evaluating 57 agents across 22 classes.

Problem

Existing agent benchmarks lack controlled, reproducible scientific tools, fail to account for cost and tool access confounders, lack standardized interfaces for general agents, and lack comprehensive baselines, making it hard to tell which scientific research agents truly advance.

Approach

Defines principles for rigorous agent benchmarking and instantiates them in AstaBench: a holistic science benchmark suite with standard MCP-based date-restricted literature search and notebook tools, time-invariant cost accounting, openness/tooling categories for reporting, and a standardized agent baselines suite.

Key results

• Asta v0 scores 52.96% overall at about $3.40 per problem, about 9 points above the best ReAct configuration (ReAct with gpt-5, 44.03%).
• The best open-weight agent scores only 11.05% (Smolagents Coder with Llama-4-Scout).
• ReAct with gpt-5-mini is the best economical configuration at 31.6% overall for about $0.04 per problem.

Surprising findings

• Most specialized agents perform worse with gpt-5 than with earlier models, while the general ReAct agent improves substantially.
• Powering a general agent with a more expensive model can lower total cost, since weaker models take more steps or loop.

Limitations

• Closed-weight frontier models dominate; open-weight agents score far lower.
• End-to-end discovery success rates remain near zero for full experiments.
• LLM-as-judge grading for open-ended tasks is still being refined.

Experimental setup

Datasets: LitQA2 ArxivDIGESTables SUPER CORE-Bench DS-1000 DiscoveryBench

Benchmarks: PaperFindingBench LitQA2-FullText LitQA2-FullText-Search ScholarQA-CS2 ArxivDIGESTables-Clean SUPER-Expert CORE-Bench-Hard DS-1000 DiscoveryBench E2E-Bench E2E-Bench-Hard

Metrics: accuracy/score cost per problem (USD) recall confidence intervals

Baselines: ReAct Smolagents Coder Asta v0 Elicit SciSpace Deep Review OpenAI Deep Research FutureHouse Falcon STORM

Builds on: Inspect HAL CodeAct ReAct

Closest papers by meaning

• SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.93)
• InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.92)
• MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.90)
• Agent-Native Research Artifacts (NeurIPS 2026 · similarity 0.90)
• FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics (NeurIPS 2026 · similarity 0.90)
• PaperBench: Evaluating AI’s Ability to Replicate AI Research (ICML 2025 · similarity 0.90)