Research map / Large language models / Agents, code and math
SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale
Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov, Alexander Golubev
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use an interactive setup agent instead of a non-interactive pipeline (ablation) | setup synthesis (Qwen3-Coder-480B) · 103-task setup subset | pass@1: 12.1 -> 25.8 | Improved |
| interactive agent with a small model (Qwen3-30B) instead of non-interactive with Qwen3-480B (ablation) | setup synthesis · 103-task setup subset | pass@10: 15.7 -> 46.1 | Improved |
| increase number of setup attempts from 1 to 10 (ablation) | mini-SWE-agent (Qwen3-480B, 32k) · 103-task setup subset | pass@k: 25.8 -> 58.8 | Improved |
| increase setup agent context length from 32k to 128k (ablation) | mini-SWE-agent (Qwen3-480B) · 103-task setup subset | pass@10: 58.8 -> 55.9 (64k) -> 62.7 (128k) | Mixed |
| use Verified+ prompt instead of Verified prompt for issue clarity (ablation) | gpt-oss-120b judge · SWE-bench Verified annotations | F1: 0.36 -> 0.50 | Improved |
| give patch and test patch to the judge (Verified-E) (ablation) | gpt-oss-120b judge · SWE-bench Verified annotations | precision: precision 0.75 -> 0.83; F1 0.36 -> 0.17 | Mixed |
| average scores across three LLM judges instead of a single model (ablation) | issue clarity filter · SWE-bench Verified annotations | F1: 0.34 -> 0.43 | Improved |
| require strict consensus of three judges (ablation) | issue clarity filter · SWE-bench Verified annotations | precision: precision 0.76 -> 0.88; recall 0.22 -> 0.06 | Mixed |
| restrict to tasks tagged Code A (clean) rather than Code B* diagnostic labels | frontier model evaluation · 60-task A vs B* sample | pass@3: Gemini 4.0 -> 34.0; GLM-4.7 6.0 -> 34.0 | Improved |
| use stricter repository filters (25 stars, 15 closed issues) for high-resource languages | data collection · repository filtering | repos requiring setup: retains about 20% of repositories while covering roughly 80% of tasks | Improved |
| rerun each task three times and keep only stable outcomes | validation · task funnel | tasks retained: 33,049 -> 32,079 tasks | No clear effect |
| generate PR-derived problem statements with leakage filtering | PR-based task expansion · 509 PR-derived tasks from SWE-bench Pro | clean fraction: 392/509 (77.0%) clean, 117/509 (23.0%) some leakage, 12/509 (2.4%) explicit leakage | Mixed |
Closest papers by meaning
- SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents (NeurIPS 2025 · similarity 0.95)
- SWE-smith: Scaling Data for Software Engineering Agents (NeurIPS 2025 · similarity 0.92)
- Training Software Engineering Agents and Verifiers with SWE-Gym (ICML 2025 · similarity 0.91)
- Toward Training Superintelligent Software Agents through Self-Play SWE-RL (ICML 2026 · similarity 0.90)
- SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution (NeurIPS 2025 · similarity 0.90)
- Kimi-Dev: Agentless Training as Skill Prior for SWE-agents (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
