Research map / Multi-agent systems and games / LLM agents
PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PatchPilot, a rule-based planning software-patching agent with a unique refinement loop that is cheaper and more stable than agent-based patchers, plus an early formal-verification add-on using CrossHair.
Problem
Existing LLM patching agents either use agent-based planning, which is high-cost and unstable, or rule-based planning, which is cheap but limited by rigid workflows that regenerate patches from scratch on failure. The paper targets resolving SWE-bench issues with a balance of efficacy, stability and
Approach
A fixed five-stage workflow (reproduction, localization, generation, validation, refinement) where failed patches are iteratively refined using validation feedback rather than regenerated, with self-reflective PoC reproduction, tool-augmented hierarchical localization, plan-then-generate with diverse prompts, and PoC-plus-functionality test ranking.
Key results
- PatchPilot resolves 45.33% (136/300) on SWE-bench Lite at $0.97/instance and 53.60% (268/500) on SWE-bench Verified at $0.99/instance with Claude-3.5-Sonnet, beating all listed open-source methods.
- Across three runs on 45 instances with GPT-4o, PatchPilot resolved 30, 32, 35 vs OpenHands 15, 20, 21, at about $12.65 vs $33.69 average run cost.
- Refinement raises the Lite resolved rate from 41.67% to 45.33%; PoC-only validation lowers it to 37.00% versus 38.7% without validation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add localization tools and review, and two-phase patch planning and generation (ablation) | PatchPilot localization and generation · SWE-bench Lite | resolved rate: 32.7% (98/300) -> 38.7% (116/300) | Helped |
| add PoC-only validation (ablation) | PatchPilot with improved localization and generation · SWE-bench Lite | resolved rate: 38.7% (116/300) -> 37.00% (111/300) | Hurt |
| add functionality tests to validation (ablation) | PatchPilot with PoC validation · SWE-bench Lite | resolved rate: 37.00% (111/300) -> 41.67% (125/300) | Helped |
| add iterative patch refinement using validation feedback (ablation) | PatchPilot with full validation · SWE-bench Lite | resolved rate: 41.67% -> 45.33% | Helped |
| use o3-mini instead of GPT-4o | PatchPilot · SWE-bench Lite subset (100 instances) | resolved rate: 19.00% -> 43.00% | Helped |
| use Claude-3.5-Sonnet instead of GPT-4o | PatchPilot · SWE-bench Lite subset (100 instances) | resolved rate: 19.00% -> 39.00% | Helped |
| use rule-based PatchPilot instead of agent-based OpenHands | PatchPilot and OpenHands · 45 common SWE-bench Lite instances, three runs | resolved instances: PatchPilot: 30, 32, and 35; OpenHands: 15, 20, and 21 | Helped |
| use Tree of Thoughts in generation (ablation) | PatchPilot · SWE-bench Lite | resolved rate | No clear effect |
| add function summaries to generation (ablation) | PatchPilot · SWE-bench Lite | resolved rate | Mixed |
| use rule-based criteria instead of an LLM judge for PoC outputs (ablation) | PatchPilot validation · SWE-bench Lite | resolved rate | Hurt |
Surprising findings
- Validation using only PoC tests lowered the resolved rate below the version without any validation, showing that fixing the PoC alone can introduce functional regressions.
- Static analysis add-ons (function summaries, rule-based PoC checks, CodeQL) did not help or were worse than LLM-based components.
Limitations
- Formal verification covered only eleven patches on SWE-bench Lite due to limits of Python verifiers.
- Stability comparison uses only 45 instances and a single model (GPT-4o), and the DeepSeek-r1 integration failed due to API issues.
- Still lags the top closed-source leaderboard entries in resolved rate.
Experimental setup
Datasets: SWE-bench Lite SWE-bench Verified
Benchmarks: SWE-bench Lite SWE-bench Verified
Metrics: resolved rate average cost per instance stability (std across runs)
Baselines: OpenHands AutoCodeRover Agentless Globant Code Fixer Agent CodeStory Midwit Agent SpecRover RepoGraph
Builds on: Agentless Reflexion Chain-of-thought Tree of Thoughts CrossHair
Closest papers by meaning
- Co-PatcheR: Collaborative Software Patching with Component-specific Small Reasoning Models (NeurIPS 2025 · similarity 0.90)
- RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.87)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.87)
- SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades (NeurIPS 2026 · similarity 0.87)
- ProgramBench: Can Language Models Rebuild Programs From Scratch? (NeurIPS 2026 · similarity 0.87)
- SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.