Research map / Multi-agent systems and games / LLM agents

From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AutoExperiment, a benchmark where agents fill in n masked functions in ML paper codebases (from MLRC replications) and must rerun experiments to match published results, interpolating from reproduction to replication.

Problem

Existing benchmarks either test reproduction (running existing code) or from-scratch replication (writing code from paper text) at a fixed difficulty; there is no benchmark that controls how much of the original code is provided when agents implement and run ML experiments from papers.

Approach

Mask n core functions from published research repositories (n=1 to 5) and have agents implement them using the paper text, the run command, and sandboxed execution, scoring by whether results are within 5% of gold code; the masking knob controls difficulty from partial reproduction to full replication.

Key results

• Claude-3.7-Sonnet reaches 36.5% pass rate at n=1 but drops to 9.6% at n=2; most models reach negligible pass rates by n=5.
• GPT-4o pass rate rises from 35.3% (Pass@1) to 48.2% (Pass@5) at n=1.
• Dynamic interaction improves GPT-4o from 8.3% (fixed agentless) to 35.3% (dynamic agent).

Surprising findings

• Without paper text access, GPT-4o performs about as well at n=1 (34.1% vs 35.3%), since missing functions are often inferable from the surrounding codebase.
• Embedding-based code retrieval (41.7%) outperforms full in-context code (36.1%), suggesting that putting all code into context degrades performance.

Limitations

• Only four ML papers and 85 functions; limited to machine learning and Python code.
• Reported results use small sample sizes (e.g., 24-36 tasks in some comparisons), and runs are resource-intensive.
• Some test cases use curtailed training runs to reduce runtime.

Experimental setup

Datasets: ITI-GEN Studying How to Efficiently and Effectively Guide Models with Explanations Cartoon Explanations of Image Classifiers Joint Multisided Exposure Fairness for Recommendation

Benchmarks: AutoExperiment

Metrics: pass rate Pass@1 Pass@5

Baselines: GPT-4o-mini Claude-3.5-Sonnet Claude-3.7-Sonnet o1 o3-mini MLAgentBench Planning-Only

Builds on: ReAct MLAgentBench SWE-bench agentless harness test-time compute scaling (budget forcing)

Closest papers by meaning

• InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.90)
• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.90)
• MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.89)
• EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.89)
• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.89)
• AI-Researcher: Autonomous Scientific Innovation (NeurIPS 2025 · similarity 0.89)