Research map / Large language models / Agents, code and math
SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SWT-Bench, a benchmark transforming SWE-Bench into issue-reproducing test generation, and evaluates LLM prompting methods and adapted Code Agents, finding agents outperform specialized test generators and generated tests can filter fixes to double precision.
Problem
Automatic generation of tests that reproduce GitHub issues in Python is underexplored; existing test generation methods are limited in scale, language coverage, and are not designed for issue reproduction with Code Agents.
Approach
Repurpose code-repair tasks as test-generation tasks: a generated test counts as successful if it fails before the golden patch and passes after (fail-to-pass), with change coverage of golden-patch lines as a finer metric; adapt existing Code Agents (SWE-Agent, aider, AutoCodeRover) by changing their instructions to write tests, and introduce a fault-tolerant diff format.
Key results
- SWE-Agent+ achieves a success rate of 18.5% on SWT-Bench-Lite, versus 14.1% for LIBRO and 9.4% for ZeroShotPlus with GPT-4.
- Filtering SWE-Agent code fixes with self-generated fail-to-pass tests raises precision from about 23.9% to 47.8% (doubling), at about 20% recall.
- Only about 20% overlap in solved instances between SWE-Bench and SWT-Bench; ideal ensemble of four test methods solves 87 instances vs 51 for the best single method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace unified diff with the custom fault-tolerant diff format (ablation) | ZeroShot test generation · SWT-Bench-Lite | success rate: 3.6 -> 9.4 | Helped |
| Instruct the agent to execute generated tests before submitting (ablation) | SWE-Agent · SWT-Bench-Lite | success rate: 15.9% to 18.5% | Helped |
| Provide the test file to change instead of BM-25 retrieved files (ablation) | ZeroShotPlus · SWT-Bench-Lite | success rate: 8.1% to 15.1% | Helped |
| Provide the golden code patch and its modified files (ablation) | ZeroShotPlus · SWT-Bench-Lite | success rate: 8.1% to 10.5% | Helped |
| Provide an incorrect code patch and its modified files (ablation) | ZeroShotPlus · SWT-Bench-Lite | success rate: 8.1% to 10.5% | Helped |
| Use Mistral Large 2 instead of GPT-4 (ablation) | SWE-Agent · SWT-Bench-Lite | success rate: 15.9 -> 16.3 | Helped |
| Use Claude 3.5 Sonnet instead of GPT-4 (ablation) | SWE-Agent · SWT-Bench-Lite | success rate: 15.9 -> 12.3 | Hurt |
| Use GPT-4o mini instead of GPT-4 (ablation) | SWE-Agent · SWT-Bench-Lite | success rate: 15.9 -> 9.8 | Hurt |
| Use Claude 3.0 Haiku instead of GPT-4 (ablation) | SWE-Agent · SWT-Bench-Lite | success rate: 15.9 -> 2.5 | Hurt |
| Use Mixtral 8x22B instead of GPT-4 (ablation) | SWE-Agent · SWT-Bench-Lite | success rate: 15.9 -> 0.7 | Hurt |
| Increase the number of libro samples (ablation) | libro · SWT-Bench-Lite | success rate | Helped |
| Increase Code Agent interaction rounds (ablation) | Code Agents · SWT-Bench-Lite | success rate | Helped |
Surprising findings
- Code repair agents designed for SWE-Bench (SWE-Agent) outperform purpose-built test generation methods such as LIBRO without test-specific design.
- Most issues were created before the GPT-4 knowledge cutoff, and the paper finds no statistically significant performance difference before vs after cutoff.
Limitations
- Limited to Python and popular GitHub repositories, which may not generalize to other languages or private code.
- Restricted to bug reproduction via adding test cases; does not measure edge case detection or global coverage increase.
- Risk of data contamination since issues are historic and may appear in LLM pretraining data.
Experimental setup
Datasets: SWE-Bench
Benchmarks: SWT-Bench SWT-Bench-Lite
Metrics: success rate fail-to-pass rate fail-to-any rate pass-to-pass rate change coverage patch well-formedness precision
Baselines: ZeroShot ZeroShotPlus PassAtFive (oracle selection) LIBRO AutoCodeRover aider SWE-Agent
Builds on: SWE-Bench SWE-Agent LIBRO AutoCodeRover aider
Closest papers by meaning
- Automated Benchmark Generation for Repository-Level Coding Tasks (ICML 2025 · similarity 0.92)
- SWE-bench Goes Live! (NeurIPS 2025 · similarity 0.91)
- TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark (ICLR 2025 · similarity 0.90)
- WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch (NeurIPS 2025 · similarity 0.89)
- SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution (NeurIPS 2026 · similarity 0.89)
- QAagent: A Multiagent System for Unit Test Generation via Natural Language Pseudocode (Student Abstract) (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.