AI Paper Atlas

Research map / Large language models / Agents, code and math

SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale

Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov, Alexander Golubev

ICML 2026 · Large language models › Agents, code and math · approach: Software engineering agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use an interactive setup agent instead of a non-interactive pipeline (ablation)setup synthesis (Qwen3-Coder-480B) · 103-task setup subsetpass@1: 12.1 -> 25.8Improved
interactive agent with a small model (Qwen3-30B) instead of non-interactive with Qwen3-480B (ablation)setup synthesis · 103-task setup subsetpass@10: 15.7 -> 46.1Improved
increase number of setup attempts from 1 to 10 (ablation)mini-SWE-agent (Qwen3-480B, 32k) · 103-task setup subsetpass@k: 25.8 -> 58.8Improved
increase setup agent context length from 32k to 128k (ablation)mini-SWE-agent (Qwen3-480B) · 103-task setup subsetpass@10: 58.8 -> 55.9 (64k) -> 62.7 (128k)Mixed
use Verified+ prompt instead of Verified prompt for issue clarity (ablation)gpt-oss-120b judge · SWE-bench Verified annotationsF1: 0.36 -> 0.50Improved
give patch and test patch to the judge (Verified-E) (ablation)gpt-oss-120b judge · SWE-bench Verified annotationsprecision: precision 0.75 -> 0.83; F1 0.36 -> 0.17Mixed
average scores across three LLM judges instead of a single model (ablation)issue clarity filter · SWE-bench Verified annotationsF1: 0.34 -> 0.43Improved
require strict consensus of three judges (ablation)issue clarity filter · SWE-bench Verified annotationsprecision: precision 0.76 -> 0.88; recall 0.22 -> 0.06Mixed
restrict to tasks tagged Code A (clean) rather than Code B* diagnostic labelsfrontier model evaluation · 60-task A vs B* samplepass@3: Gemini 4.0 -> 34.0; GLM-4.7 6.0 -> 34.0Improved
use stricter repository filters (25 stars, 15 closed issues) for high-resource languagesdata collection · repository filteringrepos requiring setup: retains about 20% of repositories while covering roughly 80% of tasksImproved
rerun each task three times and keep only stable outcomesvalidation · task funneltasks retained: 33,049 -> 32,079 tasksNo clear effect
generate PR-derived problem statements with leakage filteringPR-based task expansion · 509 PR-derived tasks from SWE-bench Proclean fraction: 392/509 (77.0%) clean, 117/509 (23.0%) some leakage, 12/509 (2.4%) explicit leakageMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy