Research map / Multi-agent systems and games / LLM agents
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces an automated, LLM-driven pipeline that mines executable Python SWE tasks from GitHub, builds the 21k-task SWE-rebench training dataset, and a continuously refreshed, decontaminated agent benchmark that reveals possible contamination on SWE-bench Verified.
Problem
Interactive, executable software engineering training data is scarce and not diverse or scalable, and static SWE benchmarks like SWE-bench Verified become contaminated as models are trained on them, making agent comparisons unreliable.
Approach
A fully automated pipeline that links issues to PRs from GitHub Archive, uses an agentless LLM (Qwen2.5-72B-Instruct) to generate and iteratively repair installation recipes, verifies tasks via execution-based FAIL_TO_PASS/PASS_TO_PASS checks, and tags tasks with LLM-predicted quality labels; dates of issues vs. model release enable contamination flagging in a fresh benchmark.</think>
Key results
- GPT-4.1 resolves 31.1% on SWE-rebench Jan 2025 but 26.7% on Mar-Apr 2025, the only model to decline noticeably between the two windows.
- DeepSeek-V3-0324 scores 39.7% on SWE-bench Verified versus 21.3% on SWE-rebench Mar-Apr 2025, suggesting possible contamination on the older benchmark.
- The fine-tuned Qwen2.5-72B quality classifier reaches 81% accuracy on task complexity labels versus 68% for the vanilla model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Enable think mode | Qwen3 models · SWE-rebench Jan and Mar–Apr 2025 | Resolved: Qwen3-235B-A22B no-think 15.2 vs think 13.7 (Jan); 16.6 vs 12.2 (Mar–Apr); Qwen3-32B no-think 13.2 vs think 11.8 (Jan); | No clear effect |
| Use Qwen3-235B-A22B no-think instead of think mode | Qwen3-235B-A22B · SWE-rebench Jan and Mar–Apr 2025 | Pass@5: 29.9 vs 29.9 (Jan); 25.7 vs 25.7 (Mar–Apr) | No clear effect |
| Evaluate on March–April tasks instead of January tasks | GPT-4.1-2025-04-14 · SWE-rebench | Resolved: 31.1 -> 26.7 | Hurt |
| Increase agentless recipe candidates from 1 to 3 | Agentless recipe generation · 18 SWE-bench tasks | Successfully configured: 6 out of 18 -> 8 out of 18 | Helped |
| Increase agentless recipe candidates from 3 to 10 | Agentless recipe generation · 18 SWE-bench tasks | Successfully configured: 8 out of 18 -> 9 out of 18 | Helped |
| Use an interactive agent instead of agentless recipe generation with 1 candidate | Installation recipe generation · 18 SWE-bench tasks | Successfully configured: 6 out of 18 -> 8 out of 18 | Helped |
| Fine-tune Qwen2.5-72B-Instruct on SWE-bench Verified annotations | Qwen2.5-72B-Instruct · Task Complexity validation set | Accuracy: 68% -> 81% | Helped |
| Fine-tune Qwen2.5-72B-Instruct on SWE-bench Verified annotations | Qwen2.5-72B-Instruct · Test Patch Correctness validation set | Accuracy: 0.60 -> 0.67 | Helped |
| Fine-tune Qwen2.5-72B-Instruct on SWE-bench Verified annotations | Qwen2.5-72B-Instruct · Issue Clarity validation set | Accuracy: 79% | Helped |
| Increase the number of changed files | DeepSeek-V3-0324 · SWE-rebench leaderboard tasks (January–July 2025) | Resolved: 28.6% ± 0.8% (1 file); 20.6% ± 0.4% (2 files); 17.5% ± 2.2% (≥ 3 files) | Hurt |
Surprising findings
- Qwen3 models perform similarly with or without think mode, and the no-think variant sometimes scores slightly higher.
- Agent-based installation configured 8/18 repositories versus 9/18 for agentless with 10 candidates, so the cheaper agentless method was chosen despite comparable yield.
Limitations
- Automated quality labels are imperfect; test-patch correctness prediction accuracy is only 67%.
- Dataset is limited to Python, and only about 31% of repositories yield a working installation recipe.
- Installation prompt validated on only 18 repositories; the evaluation scaffold is a minimal ReAct setup that may not reflect best-performing agents.
Experimental setup
Datasets: GitHub Archive SWE-bench Verified
Benchmarks: SWE-rebench (Jan 2025) SWE-rebench (Mar-Apr 2025) SWE-bench Verified
Metrics: resolved rate SEM pass@5
Baselines: SWE-bench Verified gpt-4.1-2025-04-14 DeepSeek-V3-0324 Qwen3-235B-A22B Llama-4-Maverick-Instruct Qwen2.5-72B-Instruct
Builds on: SWE-bench SWE-Gym ReAct SWE-bench Extra
Closest papers by meaning
- SWE-smith: Scaling Data for Software Engineering Agents (NeurIPS 2025 · similarity 0.91)
- SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution (NeurIPS 2026 · similarity 0.90)
- SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement (ICLR 2025 · similarity 0.90)
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (NeurIPS 2024 · similarity 0.90)
- Kimi-Dev: Agentless Training as Skill Prior for SWE-agents (ICLR 2026 · similarity 0.89)
- RepoLaunch: Automating Build and Management of Code Repositories across Languages and Platforms (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.