Research map / Large language models / Agents, code and math
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa R Kundurthy and 6 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add human-written requirements and interface to the problem statement (ablation) | GPT-5 (high) · SWE-Bench Pro public | resolve rate (%): 8.40% -> 25.9% | Improved |
| add human-written requirements and interface to the problem statement (ablation) | Claude Opus 4.1 · SWE-Bench Pro public | resolve rate (%): 8.20% -> 22.7% | Improved |
| use commercial (private startup) repositories instead of public copyleft repositories | Best models · SWE-Bench Pro commercial vs public | resolve rate (%): 25.9 -> 15.7 (GPT-5 high); 22.7 -> 17.8 (Claude Opus 4.1) | Hurt |
| use Agentless scaffold instead of SWE-Agent | SWE-Bench Pro evaluation · SWE-Bench Pro | resolve rate | Hurt |
| increase number of files edited in the reference patch | All models · SWE-Bench Pro public | resolve rate (%) | Hurt |
| use frontier models instead of open-weight models on 10+ file problems | Claude Opus 4.1, GPT-5 · SWE-Bench Pro public, 10+ files | resolve rate (%): above 10% vs near-zero | Improved |
| use Go or Python instead of JavaScript or TypeScript tasks | All models · SWE-Bench Pro public by language | resolve rate (%) | Mixed |
| judge on last 20 turns of trajectories rather than 10 or 40 (ablation) | LLM-as-judge failure analysis · Failure-mode classification | agreement with human validation | Improved |
Closest papers by meaning
- SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents (NeurIPS 2025 · similarity 0.91)
- SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution (NeurIPS 2026 · similarity 0.90)
- SWE-Marathon: Can AI Agents Autonomously Complete Ultra-Long-Horizon Software Work? (NeurIPS 2026 · similarity 0.90)
- SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? (ICLR 2025 · similarity 0.89)
- SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement (ICLR 2025 · similarity 0.89)
- SWE-smith: Scaling Data for Software Engineering Agents (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
