Research map / Multi-agent systems and games / LLM agents
SWE-smith: Scaling Data for Software Engineering Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SWE-smith, a pipeline that builds execution environments for Python repos and synthesizes 50k bug-injected SWE task instances, then fine-tunes Qwen2.5-Coder-32B on 5k Claude 3.7 trajectories to reach 40.2% on SWE-bench Verified.
Problem
Training data for software engineering agents is small, costly to curate, and requires terabytes of execution environments, limiting open-source LM development for SWE tasks.
Approach
Invert SWE-bench's collection: first build a Docker environment per repository, then synthesize bugs (LM-generated rewrites/modifications, AST procedural modifications, PR inversion, patch combination) that break existing tests, validated by test execution, and generate issue text with an LM. Environments are shared per repo, cutting storage.
Key results
- SWE-agent-LM-32B achieves 40.2% pass@1 on SWE-bench Verified, state of the art among open-weight models at the time, +33.4 points over the base Qwen 2.5 Coder 32B.
- SWE-agent-LM-32B reaches 30.7% on SWE-bench Lite, trained on 5k trajectories.
- Trajectories from PR Mirror tasks yield the highest resolve rate (9.2%) in the 7B ablation, with Procedural (8.6%) and LM Rewrite (8.8%) comparable; LM Modify is lowest (5.7%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on PR Mirror rather than LM Modify instances (ablation) | Qwen-2.5-Coder-Instruct 7B fine-tuned on SWE-smith trajectories · SWE-bench Verified | % resolved: 5.7 ($\pm1.5$) -> 9.2 ($\pm1.7)$ | Helped |
| Train on LM Rewrite rather than LM Modify instances (ablation) | Qwen-2.5-Coder-Instruct 7B fine-tuned on SWE-smith trajectories · SWE-bench Verified | % resolved: 5.7 ($\pm1.5$) -> 8.8 ($\pm1.7$) | Helped |
| Train on Procedural Modification rather than LM Modify instances (ablation) | Qwen-2.5-Coder-Instruct 7B fine-tuned on SWE-smith trajectories · SWE-bench Verified | % resolved: 5.7 ($\pm1.5$) -> 8.6 ($\pm1.8$) | Helped |
| Use LM-generated issues instead of fixed issue templates (ablation) | Qwen-2.5-Coder-Instruct 7B fine-tuned on PR Mirror trajectories · SWE-bench Verified | % resolved: 6.4 ($\pm 1.5$) -> 7.7 ($\pm 1.5$) | Helped |
| Use original PR issue text instead of LM-generated issues (ablation) | Qwen-2.5-Coder-Instruct 7B fine-tuned on PR Mirror trajectories · SWE-bench Verified | % resolved: 7.7 ($\pm 1.5$) -> 7.8 ($\pm 1.8$) | No clear effect |
| Use F2P test-based issue text instead of LM-generated issues (ablation) | Qwen-2.5-Coder-Instruct 7B fine-tuned on PR Mirror trajectories · SWE-bench Verified | % resolved: 7.7 ($\pm 1.5$) -> 7.3 ($\pm 1.9$) | No clear effect |
| Train on higher-difficulty-score trajectories instead of score-2 trajectories (ablation) | Qwen-2.5-Coder-Instruct 7B · SWE-bench Verified | Pass@1: 12.4\%, 10.8\%, 13.6\%, and 12.2\% | Mixed |
| Increase the number of training repositories (ablation) | SWE-agent training on Procedural Modification trajectories · SWE-bench | performance | Helped |
| Specialize training data to SymPy rather than sampling 100 repositories (ablation) | Qwen-2.5-Coder-Instruct 7B · SymPy SWE-bench Verified subset | % resolved: 13.6\% -> 21.2\% | Helped |
| Further fine-tune SWE-agent-LM-32B on SymPy trajectories (ablation) | SWE-agent-LM-32B · SymPy SWE-bench Verified subset | % resolved: 33.3\% -> 42.4\% | Helped |
| Increase the number of SWE-smith training trajectories | SWE-agent · SWE-bench | performance | Helped |
Surprising findings
- Higher task difficulty did not correlate with better downstream student performance despite lower expert resolve rates on hard tasks.
- LM-generated issue text performed comparably to original GitHub issue text for training.
Limitations
- Pipeline is Python-centric, relying on the Python ast library for transformations.
- Only fine-tuning via rejection sampling is explored; RL-based reasoning training is not studied.
- Repetitive action loops in SWE-agent-LM-32B cause many failures (more than half of unresolved instances hit cost/step limits).
Experimental setup
Datasets: SWE-smith SWE-bench Verified SWE-bench Lite SWE-bench Multilingual
Benchmarks: SWE-bench Verified SWE-bench Lite SWE-bench Multilingual
Metrics: pass@1 % resolved
Baselines: SWE-gym-32B R2E-Gym-32B SWE-fixer-72B Lingma-SWE-GPT-72B Qwen3-235B-A22B Llama3-SWE-RL-70B Claude 3.7 Sonnet Claude 3.5 Sonnet GPT-4o
Builds on: SWE-bench SWE-agent ReAct rejection sampling fine-tuning
Closest papers by meaning
- SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents (NeurIPS 2025 · similarity 0.91)
- Training Software Engineering Agents and Verifiers with SWE-Gym (ICML 2025 · similarity 0.90)
- SWE-bench: Can Language Models Resolve Real-world Github Issues? (ICLR 2024 · similarity 0.90)
- SWE-bench Goes Live! (NeurIPS 2025 · similarity 0.89)
- MLE-Smith: Scaling MLE Tasks with Automated Multi-agent Pipeline (ICLR 2026 · similarity 0.89)
- SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads? (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.