Research map / Multi-agent systems and games / LLM agents

EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces EvoCode-Bench, a 26-task, 227-round benchmark for persistent multi-turn coding with cumulative executable tests and fail-stop scoring, and evaluates 13 coding agents showing single-round scores overstate persistent reliability.

Problem

Existing coding benchmarks evaluate one specification and one final assessment, not whether coding agents can maintain a working, evolving codebase as requirements change over multiple rounds.

Approach

A persistent-workspace multi-round benchmark where each round's cumulative verifier checks all active requirements, scored with fail-stop MT@4 (four attempts) and a reference-fast-forwarded single-round SR; the gap between them reveals accumulated-state reliability and reranks agents.

Key results

• Opus-4.7 leads with 54.0 MT@4 and 76.7 SR; Opus-4.6 has the highest SR (78.9) but only 44.0 MT@4 (third).
• SR exceeds MT@4 by 22-40 points for most agents; top-to-lower tier MT@4 ratio is 5.9x vs 2.3x under SR.
• Round-level MT@4 pass rate falls from 46.7 at round 1 to 21.3 at round 5 and 7.7 at round 10.

Surprising findings

• The highest single-round (SR) scorer, Opus-4.6, ranks only third in persistent multi-turn execution.
• 57% of failed multi-turn rounds are solvable under SR from a reference-completed state.

Limitations

• Only 26 tasks; scaling constrained by manual authoring and verification cost.
• Fail-stop scoring does not measure recovery and may underestimate agents capable of repair.
• Only covers tasks verifiable by executable tests; subjective design and code quality not evaluated.

Experimental setup

Datasets: EvoCode-Bench tasks (26 multi-round tasks, 227 rounds)

Benchmarks: EvoCode-Bench

Metrics: MT@4 SR Comp Avg. Turns Output Tok. (K)

Baselines: SWE-EVO SlopCodeBench SWE-bench Terminal-Bench AppWorld

Builds on: Harbor Terminus-2 SWE-EVO SlopCodeBench

Closest papers by meaning

• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.93)
• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.91)
• RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.89)
• Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases (NeurIPS 2026 · similarity 0.89)
• From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking (ICLR 2026 · similarity 0.89)
• Huxley-G\"odel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine (ICLR 2026 · similarity 0.89)