Research map / Multi-agent systems and games / LLM agents
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the agent-computer interface (ACI) and SWE-agent, an LM agent with a custom file viewer, editor, search, and linting-guarded edit commands that resolves 12.47% of SWE-bench test tasks with GPT-4 Turbo and reaches 87.7% pass@1 on HumanEvalFix.
Problem
LM agents struggle to reliably perform software engineering tasks when given raw Linux shell access, as the shell's commands are designed for humans and are inefficient, verbose, and error-prone for language models.
Approach
Design a purpose-built interface layer (ACI) for LM agents with simple, compact commands (file viewer, search, edit with line numbers), concise environment feedback, and guardrails such as linting that rejects syntax-breaking edits, without changing the LM's weights.
Key results
- SWE-agent with GPT-4 Turbo resolves 12.47% (286/2294) of SWE-bench test tasks, versus 3.8% for the prior best RAG system.
- On SWE-bench Lite, SWE-agent w/ GPT-4 Turbo reaches 18.00%, compared with 11.00% for the Shell-only agent.
- SWE-agent achieves 87.7% pass@1 on HumanEvalFix Python, well above GPT-4 at 47.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the SWE-agent ACI instead of a Shell-only interface | GPT-4 Turbo agent · SWE-bench Lite test | % Resolved: 11.00 -> 18.00 | Helped |
| Use the SWE-agent ACI instead of RAG | Claude 3 Opus · SWE-bench test | % Resolved: 3.79 -> 10.46 | Helped |
| Add linting guardrails to the edit action (ablation) | SWE-agent · SWE-bench Lite test | % Resolved: 15.0 -> 18.0 | Helped |
| Add the edit action (ablation) | SWE-agent · SWE-bench Lite test | % Resolved: 10.3 -> 18.0 | Helped |
| Use summarized search instead of iterative search (ablation) | SWE-agent · SWE-bench Lite test | % Resolved: 12.0 -> 18.0 | Helped |
| Add search tools (ablation) | SWE-agent · SWE-bench Lite test | % Resolved: 15.7 -> 18.0 | Helped |
| Use a 100-line viewer window instead of a 30-line window (ablation) | SWE-agent · SWE-bench Lite test | % Resolved: 14.3 -> 18.0 | Helped |
| Use a 100-line viewer window instead of showing the full file (ablation) | SWE-agent · SWE-bench Lite test | % Resolved: 12.7 -> 18.0 | Helped |
| Collapse older observations and retain the last 5 observations (ablation) | SWE-agent · SWE-bench Lite test | % Resolved: 15.0 -> 18.0 | Helped |
| Add a demonstration (ablation) | SWE-agent · SWE-bench Lite test | % Resolved: 16.3 -> 18.0 | Helped |
Surprising findings
- A standard search interface (iterative, one result at a time) performed worse than having no search tool at all (12.0% vs 15.7%).
- Successful runs finish earlier and cheaper, suggesting larger budgets are unlikely to substantially raise performance.
Limitations
- Open-source models such as Llama 3 and DeepSeek Coder performed poorly in the agent setting, limiting generality.
- Ablations and design decisions were developed on a small hand-picked development set and tuned for GPT-4 Turbo.
- Cost is higher than RAG (8-13x on Lite).
Experimental setup
Datasets: SWE-bench SWE-bench Lite HumanEvalFix
Benchmarks: SWE-bench SWE-bench Lite HumanEvalFix
Metrics: % Resolved (pass@1) $ Avg. Cost pass@k
Baselines: RAG (BM25 retrieval, non-interactive) Shell-only agent CodeLLaMa-instruct-13B GPT-4 DeepseekCoder-CodeAlpaca-6.7B WaveCoder-DS-6.7B
Builds on: ReAct InterCode SWE-bench
Closest papers by meaning
- SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents (NeurIPS 2025 · similarity 0.90)
- TOM-SWE: User Mental Modeling For Software Engineering Agents (ICML 2026 · similarity 0.89)
- SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement (ICLR 2025 · similarity 0.89)
- Training Software Engineering Agents and Verifiers with SWE-Gym (ICML 2025 · similarity 0.88)
- Programming with Pixels: Can Computer-Use Agents do Software Engineering? (ICLR 2026 · similarity 0.88)
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.