Research map / Multi-agent systems and games / LLM agents
MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically analyzes why LLMs fail at GitHub issue resolution and proposes MAGIS, a four-agent (Manager, Repository Custodian, Developer, QA Engineer) framework that raises the SWE-bench resolved rate to 13.94% with GPT-4.
Problem
LLMs resolve very few GitHub issues at repository level, since locating files and lines to modify and handling code changes across a large repository is hard.
Approach
A role-based multi-agent workflow inspired by GitHub Flow: a custodian agent filters candidate files with a memory-based LLM relevance check, a manager decomposes the issue into file-level tasks and recruits developer agents, developers produce line-range-localized edits, and a QA agent reviews each change iteratively.
Key results
- MAGIS resolves 13.94% of SWE-bench issues on the 25% subset, versus 1.74% for GPT-4 directly (about eight-fold).
- Claude-2 reaches 4.88% resolved, the strongest baseline in the main table, so MAGIS more than doubles it.
- On SWE-bench Lite, MAGIS achieves 25.33% resolved, above AutoCodeRover (16.11% average) and SWE-Agent (18.00%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MAGIS instead of direct GPT-4 | GPT-4 · SWE-bench 25% subset | resolved ratio: 1.74 -> 13.94 | Helped |
| add QA Engineer review (ablation) | MAGIS · SWE-bench 25% subset | resolved ratio: 10.63 -> 13.94; 8.71 -> 10.63 | Helped |
| add pull-request comment hints (ablation) | MAGIS · SWE-bench 25% subset | resolved ratio: 10.28 -> 13.94; 8.71 -> 10.28 | Helped |
| add QA Engineer review (ablation) | MAGIS · SWE-bench Lite | resolved ratio: 23.33% -> 25.33%; 16.00% -> 16.67% | Helped |
| add pull-request comment hints (ablation) | MAGIS · SWE-bench Lite | resolved ratio: 16.67% -> 25.33%; 16.00% -> 25.33% | Helped |
Surprising findings
- Line-locating coverage correlated with resolution only for Claude-2; for GPT-4 and GPT-3.5 the sample of successes was too small for significance.
- Number of files and functions modified showed strongly negative correlation with resolution for GPT-4 and GPT-3.5.
Limitations
- Evaluation limited to 12 Python repositories in SWE-bench, so generality to other languages and paradigms is unverified.
- Results on a 25% subset of SWE-bench and a comparison with Devin on only 140 overlapping instances; comparisons are not fully equitable.
- Prompt design may bias results; the paper relies on fixed templates.
Experimental setup
Datasets: SWE-bench
Benchmarks: SWE-bench SWE-bench Lite
Metrics: % applied % resolved recall vs file number line locating coverage ratio
Baselines: GPT-3.5 GPT-4 Claude-2 SWE-Llama 7b SWE-Llama 13b AutoCodeRover SWE-Agent Devin
Builds on: BM25 MetaGPT ChatDev SWE-bench
Closest papers by meaning
- GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.87)
- RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving (NeurIPS 2025 · similarity 0.87)
- Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving (NeurIPS 2025 · similarity 0.86)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.86)
- Nemotron-CORTEXA: Enhancing LLM Agents for Software Engineering Tasks via Improved Localization and Solution Diversity (ICML 2025 · similarity 0.86)
- SWE-bench: Can Language Models Resolve Real-world Github Issues? (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.