Research map / Multi-agent systems and games / LLM agents

Agent-Native Research Artifacts

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes the Agent-Native Research Artifact (ARA), an executable, layered research package (logic, code, exploration graph, evidence) with a live manager, compiler, and seal-based review, and evaluates it on PaperBench and RE-Bench against PDF+repo baselines.

Problem

Published papers compress branching research into linear narratives, dropping failed experiments and implementation details, which hinders AI agents that must understand, reproduce, and extend published work.

Approach

Recast the primary research object as an agent-operable file-system package with four layers (scientific logic, executable code with specifications, a branching exploration graph including dead ends, and evidence bound to claims), with papers as compiled views, plus tooling to create, compile, and verify such artifacts.

Key results

• Understanding accuracy rises from 72.4% (baseline) to 93.7% (ARA) across 450 paired questions.
• Difficulty-weighted reproduction success on 15 PaperBench papers rises from 57.4% to 64.4% (win/tie/loss 8/5/2).
• On RE-Bench extension tasks, ARA agents reach useful moves earlier, but with Sonnet 4.6 the paper agent ends ahead on triton_cumsum and restricted_mlm; with Sonnet 4.5 the ARA agent trails on both.

Surprising findings

• Failed runs account for 90.2% of total dollar cost in RE-Bench agent runs, with a median failed-to-success token ratio of 113x.
• Preserved failure traces can constrain a more capable agent, causing it to underperform a paper-only baseline on some tasks.

Limitations

• Reproduction results rely on a single base model (Claude Sonnet 4.6) and one ARA agent fabricated results in one run.
• The extension experiment covers only 5 RE-Bench tasks and the artifact's value depends on agent capability.
• Physical-world experimental research falls outside the stated scope.

Experimental setup

Datasets: PaperBench RE-Bench METR eval-analysis-public METR MALT transcripts

Benchmarks: PaperBench RE-Bench

Metrics: question-answering accuracy difficulty-weighted reproduction success rate best score on extension tasks detection rate of injected errors

Baselines: PDF plus GitHub repository baseline paper agent (PDF-based writeup)

Builds on: PaperBench RE-Bench FAIR principles RO-Crate Nanopublications

Closest papers by meaning

• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.90)
• InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.89)
• Agentic AI Scientists Are Not Built For Autonomous Scientific Discovery (NeurIPS 2026 · similarity 0.88)
• SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.88)
• ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence (NeurIPS 2026 · similarity 0.88)
• From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking (ICLR 2026 · similarity 0.88)