Research map / Multi-agent systems and games / LLM agents
RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RoadmapBench, a 115-task benchmark of multi-target long-horizon feature implementation tasks derived from real open-source version upgrades across 17 repos and 5 languages, and evaluates 13 frontier models under OpenHands and Terminus 2.
Problem
Existing coding benchmarks are mostly short-horizon single-issue bug fixes in Python with binary pass/fail outcomes, failing to measure long-horizon, multi-target software development at real engineering scale.
Approach
Converts real version-to-version upgrades into multi-target roadmap instructions with per-target test suites, a weighted completion score for partial progress, and a static-validation plus rollout-based attribution pipeline that separates task defects from model failures.
Key results
- Claude-Opus-4.7 resolves 39.1% of tasks under OpenHands, the best of 13 models; the weakest (Seed-2.0-Pro) resolves 5.2%.
- Completion Score is consistently higher than resolved rate, e.g. Claude-Opus-4.6 has 32.2% resolved and 0.627 completion score.
- Resolved rate declines monotonically with files changed, lines changed, and number of subtasks; Kimi-K2.5 drops to 0% at 7+ subtasks.
Surprising findings
- OpenHands gives higher resolved rates than Terminus 2 for most models, though Claude-Opus-4.6 is nearly scaffold-invariant (31.3% vs 32.2%).
- Claude-Opus-4.7 and GLM-5.1 use similar step budgets (140 vs 163) yet differ by more than 20 points in resolved rate.
Limitations
- Evaluated with only two agent scaffolds; results are scaffold-sensitive.
- Tests check behavior but not code quality or maintainability.
- Covers only a limited subset of language ecosystems and frameworks.
Experimental setup
Datasets: real open-source version upgrade pairs from 17 repositories
Benchmarks: RoadmapBench
Metrics: resolved rate completion score average turns output tokens per-domain resolved rate
Baselines: SWE-bench Verified SWE-bench Pro FeatureBench TerminalBench SWE-EVO NL2Repo
Builds on: SWE-bench Harbor OpenHands
Closest papers by meaning
- SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.92)
- NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.91)
- GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.91)
- SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades (NeurIPS 2026 · similarity 0.90)
- EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.89)
- Automated Benchmark Generation for Repository-Level Coding Tasks (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.