Research map / Multi-agent systems and games / LLM agents

SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SWE-Chain, a benchmark of chained release-level Python package upgrades built with a divide-and-conquer spec synthesis pipeline (DecompSynth), and evaluates nine frontier agent configurations under a Build+Fix regime.

Problem

Existing coding-agent benchmarks rarely evaluate continuous maintenance across package releases, where earlier changes carry over to later versions and upgrade specifications from raw release notes and issues are noisy.

Approach

Chain consecutive release upgrades so each step builds on the agent's own prior codebase, with upgrade specifications synthesized by aligning maintainer release notes with gold code diffs (DecompSynth) and evaluated via upgrade-related tests plus regression checks on preserved behavior.

Key results

• Average Build+Fix resolving 44.8%, precision 65.4%, F1 50.2% across nine agent-model configurations.
• Claude-Opus-4.7 (Claude Code) leads with 60.8% resolving, 80.6% precision, 68.5% F1 under Build+Fix.
• Chain-level average resolving ranges from 23.3% to 68.5%, showing discriminative difficulty across chains.

Surprising findings

• Build+Fix mainly improves precision (54.0% to 65.4%) while resolving rises only modestly (39.7% to 44.8%).
• Raw release notes with linked issues/PRs as specifications give very low precision (about 9% on pytest 8.0 for Claude-Opus-4.7), while adding expectations and constraints raises it sharply.

Limitations

• Specification granularity affects results; higher-granularity (grounded/oracle-like) specs may be closer to oracle guidance than realistic workflows.
• Specifications are synthesized by GPT-5.4 and some manual quality review is involved, which may introduce bias.
• Only 9 Python packages and 12 chains, limiting coverage of ecosystems.

Experimental setup

Datasets: release notes and GitHub issues/PRs from 9 Python packages git code and test diffs between consecutive versions

Benchmarks: SWE-Chain

Metrics: resolving rate precision F1-score final passing

Baselines: SWE-EVO SWE-CI SlopCodeBench EvoClaw Terminal-Bench

Builds on: SWE-Bench SWE-EVO Aider pass-rate multi-attempt BigCodeBench-style oracle specification

Closest papers by meaning

• Automated Benchmark Generation for Repository-Level Coding Tasks (ICML 2025 · similarity 0.90)
• RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.90)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.89)
• SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution (NeurIPS 2026 · similarity 0.89)
• SWE-bench Goes Live! (NeurIPS 2025 · similarity 0.87)
• SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.87)