Research map / Large language models / Agents, code and math
SWE-bench: Can Language Models Resolve Real-world Github Issues?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SWE-bench, a benchmark of 2,294 real GitHub issue-PR tasks from 12 Python repos, and SWE-Llama, CodeLlama fine-tuned on repository-editing data, showing that top LMs resolve only a small fraction of issues.
Problem
Existing code benchmarks are self-contained and saturated, so they do not measure whether language models can resolve real-world software engineering issues that require multi-file, repository-scale edits.
Approach
Build an execution-validated benchmark from merged GitHub pull requests that resolve issues and add tests (fail-to-pass tests), so models must produce patches across large codebases; also release a training split and fine-tune CodeLlama-Python into SWE-Llama repository editors.
Key results
- Claude 2 resolves 1.96% of SWE-bench tasks with BM25 retrieval (best in the main BM25 table at 13k context), while Claude 3 Opus reaches 3.79% in the later BM25 comparison table.
- In the oracle retrieval setting, Claude 2 resolves 4.8% of instances, and SWE-Llama 13b performs comparably at about 91 resolved instances versus Claude 2's 110.
- Resolve rates drop as context grows: Claude 2 falls from 1.96% at 13k to 1.22% at 50k BM25 context, and SWE-Llama 13b reaches 0% at 50k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Collapse oracle-retrieved files to edited lines plus ±15 lines of buffer (ablation) | GPT-4 · SWE-bench, oracle retrieval | % Resolved: 1.3% -> 3.4% | Helped |
| Collapse oracle-retrieved files to edited lines plus ±15 lines of buffer (ablation) | Claude 2 · SWE-bench, oracle retrieval | % Resolved: 4.8% -> 5.9% | Helped |
| Use BM25 retrieval with larger maximum context limits (ablation) | Claude 2 · SWE-bench | % Resolved: 1.96 -> 1.87 -> 1.22 | Hurt |
| Use BM25 retrieval with larger maximum context limits (ablation) | SWE-Llama 7b · SWE-bench | % Resolved: 0.70 -> 0.31 -> 0.00 | Hurt |
| Use BM25 retrieval with larger maximum context limits (ablation) | SWE-Llama 13b · SWE-bench | % Resolved: 0.70 -> 0.48 -> 0.00 | Hurt |
| Use oracle retrieval instead of BM25 retrieval | Claude 2 · SWE-bench | % Resolved: 1.96% -> 4.8% | Helped |
| Regenerate entire files rather than generate patch files (ablation) | Claude 2 · SWE-bench, oracle retrieval | % Resolved: 4.8% -> 2.2% | Hurt |
| Use whole-file generation rather than patches on the shorter half of tasks (ablation) | Claude 2 · SWE-bench, oracle retrieval, shorter half by input tokens | % Resolved: 7.8% -> 3.9% | Hurt |
| Increase BM25 maximum context length (ablation) | Models using BM25 retrieval · SWE-bench | % Resolved | Hurt |
| Fine-tune using oracle-file context, then evaluate with BM25 context | SWE-Llama · SWE-bench | % Resolved | Hurt |
| Use more context files that are not expected to be edited | SWE-Llama · SWE-bench with BM25 context | % Resolved | Hurt |
Surprising findings
- Models perform worse when asked to regenerate entire files than when generating patch files (Claude 2 2.2% vs 4.8%).
- Fine-tuned SWE-Llama performs poorly with BM25-retrieved context despite strong oracle-context performance, indicating sensitivity to context distribution shift.
Limitations
- Only Python repositories are covered.
- Evaluation relies solely on execution-based tests, which may miss code quality issues such as readability and efficiency.
- Baselines use simple retrieval and direct prompting rather than agent-based approaches.
Experimental setup
Datasets: SWE-bench SWE-bench-train
Benchmarks: SWE-bench SWE-bench Lite
Metrics: % resolved % apply
Baselines: Claude 2 ChatGPT-3.5 GPT-4 Claude 3 Opus GPT-4-turbo SWE-Llama 7b SWE-Llama 13b
Builds on: CodeLlama LoRA BM25 HumanEval
Closest papers by meaning
- SWE-bench Goes Live! (NeurIPS 2025 · similarity 0.92)
- SWE-smith: Scaling Data for Software Engineering Agents (NeurIPS 2025 · similarity 0.90)
- SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads? (ICML 2026 · similarity 0.90)
- Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving (NeurIPS 2025 · similarity 0.89)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.88)
- ProgramBench: Can Language Models Rebuild Programs From Scratch? (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.