Research map / Multi-agent systems and games / LLM agents

InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces InnovatorBench, a 20-task benchmark of end-to-end LLM research tasks, and ResearchGym, a long-horizon, distributed agent environment, evaluated with a ReAct agent on four frontier LLMs.

Problem

Existing AI research agent benchmarks test narrow skills in simplified, resource-constrained settings and mostly reproduce known results, rather than assessing end-to-end, innovation-oriented LLM research over long horizons.

Approach

A Kaggle-style benchmark of 20 LLM research tasks (data construction, filtering, augmentation, loss design, reward design, scaffold construction) with hidden reference solutions and external scoring, paired with an environment supporting asynchronous execution, multi-machine control, and snapshots.

Key results

• Claude Sonnet 4 achieves the highest weighted average final score of 24.01 on InnovatorBench, versus 12.04 for GPT-5, 11.85 for GLM-4.5 and 5.35 for Kimi-K2.
• GPT-5 scores 60.07 on Scaffold Construction, the best domain result for any model, while scoring near 0 on several algorithm tasks.
• Agents need over 11 hours to reach peak performance on InnovatorBench versus about 1.75 hours on PaperBench (about 6.5x longer test-time scaling).

Surprising findings

• Providing the ground-truth hint lowered Claude Sonnet 4's weighted average final score from 24.01 to 13.88, with data tasks hurt most.
• Frontier agents often fail through impatience (killing training early), resource contention, and choosing slower inference libraries such as Transformers over vLLM.

Limitations

• Limited task diversity: only 20 tasks from 14 papers, with the authors acknowledging narrow coverage.
• Results depend heavily on the agent model and its tool-use reliability, so generalization across models is uneven.
• Scoring is calibrated between baseline and reference solutions, and single-run variance with few evaluations may affect comparability.

Experimental setup

Datasets: DatasetResearch tasks DAPO math data LIMO NuminaMath SuperGPQA Search-R1 data

Benchmarks: InnovatorBench

Metrics: final score best score execution time cost

Baselines: PaperBench Claude Sonnet 4 GPT-5 GLM-4.5 Kimi-K2

Builds on: ReAct OpenHands LlamaFactory Verl PaperBench

Closest papers by meaning

• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.92)
• MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.90)
• FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics (NeurIPS 2026 · similarity 0.90)
• From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking (ICLR 2026 · similarity 0.90)
• SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.90)
• SciAgentGym: Benchmarking Multi-Step Scientific Tool-Use in LLM Agents (ICML 2026 · similarity 0.90)