Research map / Multi-agent systems and games / LLM agents
Can Agent Fix Agent Issues?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a taxonomy of 201 real-world agent-system issues and a 50-task reproducible benchmark (AgentIssue-Bench), then shows open SE agents (Agentless, AutoCodeRover, SWE-agent) resolve only 0.67%–4.67% of them.
Problem
Automated resolution of bug reports and feature requests in LLM-based agent systems is unexplored; it is unclear whether SE agents that work on traditional software can fix issues in agent codebases.
Approach
Empirically characterize agent-specific issues (6 categories, 20 subcategories) from GitHub, manually reproduce 50 of them into executable Docker tasks with failure-triggering tests, and evaluate existing SE agents with plausible and human-verified correct resolution rates.
Key results
- Best correct resolution rate is 4.67% (AutoCodeRover with Claude-3.5-Sonnet); range across agents and LLMs is 0.67%–4.67%.
- Agent issue resolution rates are far below SWE-bench Lite rates (23.20%–50.80%) for the same SE agents.
- Utility and dependency issues are the most-resolved categories, while LLM operation and LLM-provider incompatibility issues remain largely unsolved.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Claude-3.5-S instead of GPT-4o | Agentless · AgentIssue-Bench | correctly resolved (%): 3.33 -> 4.00 | Helped |
| Use Claude-3.5-S instead of GPT-4o | AutoCodeRover · AgentIssue-Bench | correctly resolved (%): 1.33 -> 4.67 | Helped |
| Use Claude-3.5-S instead of GPT-4o | SWE-agent · AgentIssue-Bench | correctly resolved (%): 0.67 -> 2.00 | Helped |
Surprising findings
- Each SE agent uniquely solves 1–2 issues, and no single issue is solved by all agents, indicating complementary capabilities.
- Average cost per issue ranges from $0.05 to $1.15, similar to costs on traditional software issues.
Limitations
- Benchmark is small (50 tasks) because reproduction of agent issues is very labor-intensive (~500 person-hours).
- Only reactive, user-reported issues are covered; proactive maintenance and performance issues are excluded.
- Only three SE agents and two LLMs are evaluated; results have high variance with small counts.
Experimental setup
Datasets: 201 GitHub agent issues from 18 agent systems SWE-bench Lite
Benchmarks: AgentIssue-Bench
Metrics: plausible resolution rate correct resolution rate file-level localization function-level localization cost
Baselines: Agentless AutoCodeRover SWE-agent
Builds on: SWE-bench Agentless SWE-agent AutoCodeRover
Closest papers by meaning
- SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents (NeurIPS 2025 · similarity 0.86)
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (NeurIPS 2024 · similarity 0.86)
- DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems (ICLR 2026 · similarity 0.85)
- OrcaLoca: An LLM Agent Framework for Software Issue Localization (ICML 2025 · similarity 0.85)
- Why Do Multi-Agent LLM Systems Fail? (NeurIPS 2025 · similarity 0.85)
- MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.