Research map / Large language models / Agents, code and math
SWE-bench Goes Live!
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SWE-bench-Live, a live-updatable, Docker-reproducible issue-resolving benchmark of 1,319 real GitHub issues from 93 Python repos, built by an agentic automated pipeline (RepoLaunch), and evaluates agent-LLM pairs on it.
Problem
Existing issue-resolving benchmarks such as SWE-bench are static, cover few repositories, and rely on heavy manual effort for instance construction and environment setup, risking contamination and overfitting.
Approach
An LLM-driven agentic pipeline (RepoLaunch) that automatically identifies setup files, selects base Docker images, installs dependencies with a time-machine package index, and validates FAIL_TO_PASS/PASS_TO_PASS transitions, enabling monthly live updates of a benchmark built from recent issues.
Key results
- Best configuration OpenHands with Claude 3.7 Sonnet resolves 19.25% of the full SWE-bench-Live set (17.67% on Lite).
- The same OpenHands with Claude 3.7 Sonnet setup resolves 43.20% on SWE-bench Verified, more than twice its SWE-bench-Live score.
- Single-file patches under five lines are solved about 48% of the time, while patches touching seven or more files are never solved.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate OpenHands + Claude 3.7 Sonnet on SWE-bench-Live instead of SWE-bench Verified | OpenHands + Claude 3.7 Sonnet · SWE-bench-Live vs. SWE-bench Verified | Resolved rate (%): 43.20% -> 19.25% | Hurt |
| Use repositories previously included in SWE-bench rather than repositories not previously used there | Best-performing agent-model pair · SWE-bench-Live repository groups | Resolved (%): 18.89 -> 22.96 | Helped |
| Increase patch scope to three or more files or more than one hundred lines | Current agents · SWE-bench-Live full | Resolved rate: below ten per-cent | Hurt |
| Keep the gold fix to one file and fewer than five lines | Current agents · SWE-bench-Live full | Resolved rate: 48% | Helped |
| Evaluate on larger repositories exceeding five-hundred files rather than smaller repositories under one hundred files and twenty-thousand LOC | Current agents · SWE-bench-Live repository difficulty | Resolved rate: above twenty per-cent; rarely exceed five per-cent | Hurt |
| Use newer issue creation periods instead of older periods | OpenHands + Claude 3.7 Sonnet · SWE-bench-Live full, quarterly issue cohorts | Resolved rate | No clear effect |
Surprising findings
- Agent-LLM pairs score substantially lower on SWE-bench-Live than on SWE-bench Verified under identical settings, suggesting overfitting to static benchmarks.
- Resolved rate does not clearly decline for more recent issues across 2024Q1 to 2025Q1.
Limitations
- Only Python repositories are included; other languages are future work.
- Single run per configuration due to API budget, so LLM randomness is not averaged out.
- Agentless omits the regression-test-based reranking stage.
Experimental setup
Datasets: GitHub issues and pull requests from 93 Python repositories created 2024 to April 2025
Benchmarks: SWE-bench-Live SWE-bench Verified
Metrics: Resolved Rate Patch Apply Rate Localization Success Rate
Baselines: SWE-bench Verified OpenHands SWE-Agent Agentless Multi-SWE-bench OmniGIRL
Builds on: SWE-bench ReAct OpenHands SWE-Agent Agentless
Closest papers by meaning
- SWE-bench: Can Language Models Resolve Real-world Github Issues? (ICLR 2024 · similarity 0.92)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.91)
- Automated Benchmark Generation for Repository-Level Coding Tasks (ICML 2025 · similarity 0.90)
- SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads? (ICML 2026 · similarity 0.90)
- Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving (NeurIPS 2025 · similarity 0.89)
- SWE-smith: Scaling Data for Software Engineering Agents (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.