Research map / Large language models / Agents, code and math
Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Multi-SWE-bench, a 1,632-instance human-verified issue-resolving benchmark across seven non-Python languages, evaluates nine LLMs with three adapted agent methods, and launches the Multi-SWE-RL community dataset of 4,723 instances.
Problem
Existing issue-resolving benchmarks such as SWE-bench focus almost exclusively on Python, so it is unclear whether LLM agents generalize to other widely used programming languages.
Approach
A five-phase pipeline (repo selection, PR crawling, Dockerized environment setup, full-test-suite status-transition filtering, and dual-annotator manual verification) builds a multilingual benchmark, and the Python-oriented agents (Agentless, SWE-agent, OpenHands) are adapted to seven other languages to evaluate them.
Key results
- Across methods, models resolve Python issues at much higher rates than other languages; for example, Claude-3.7-Sonnet with MopenHands reaches 52.20% on Python but 2.23% on TypeScript and 0% on hard-level issues for most agents.
- MopenHands achieves the highest resolved rate in five of seven non-Python languages, per the paper's summary of Table 3.
- The Multi-SWE-RL initial release contains 4,723 containerized instances across 76 repositories and seven languages.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MopenHands rather than MagentLess or MSWE-agent | Issue-resolving methods evaluated with nine LLMs · Multi-SWE-bench languages | resolved rate (%) | Mixed |
| Use Claude-3.7-Sonnet rather than lower-performing evaluated models | Issue-resolving methods · Multi-SWE-bench | resolved rate (%) | Mixed |
| Resolve easier rather than harder issues | LLM-based agents · Multi-SWE-bench difficulty levels | resolved rate (%) | Hurt |
| Handle longer rather than shorter fix patches | MagentLess, MSWE-agent, and MopenHands · Multi-SWE-bench | resolved rate (%): approximately 50% lower | Hurt |
| Modify more files rather than a single file | MagentLess, MSWE-agent, and MopenHands · Multi-SWE-bench | resolved rate (%) | Hurt |
| Use a more flexible agent workflow rather than MagentLess's fixed workflow | Issue-resolving methods · Multi-SWE-bench | resolved rate (%) | Helped |
| Address bug fixes rather than new features or feature optimizations | MagentLess, MSWE-agent, and MopenHands · Multi-SWE-bench issue types | resolved rate (%) | Helped |
| Use a repository with lower language entropy | Issue-resolving agents · Multi-SWE-bench repositories | resolved rate (%) | Helped |
| Use longer issue descriptions | MagentLess · Python, Java, and TypeScript in Multi-SWE-bench | resolved rate (%) | Helped |
| Increase issue-description length | Issue-resolving agents · Python and Go in Multi-SWE-bench | resolved rate (%) | Mixed |
| Prioritize accurate issue localization over code editing | MagentLess, MSWE-agent, and MopenHands · Multi-SWE-bench | resolved rate (%) | Mixed |
Surprising findings
- Resolved rates approach zero on hard (over one hour human effort) issues for existing LLMs and agents.
- Some certain easy Rust instances involve large-scale edits due to repetitive, pattern-consistent changes.
Limitations
- Agent methods were originally designed for Python and only partially adapted, so results may reflect method bias.
- Candidate patch selection (regression and reproduction testing) was removed from Agentless adaptation.
- Evaluation covers only a limited set of models and three agent methods.
Experimental setup
Datasets: Multi-SWE-bench Multi-SWE-RL
Benchmarks: Multi-SWE-bench
Metrics: resolved rate success location average cost
Baselines: Agentless SWE-agent OpenHands
Builds on: SWE-bench Agentless SWE-agent OpenHands
Closest papers by meaning
- SWE-bench: Can Language Models Resolve Real-world Github Issues? (ICLR 2024 · similarity 0.89)
- SWE-bench Goes Live! (NeurIPS 2025 · similarity 0.89)
- MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution (NeurIPS 2024 · similarity 0.86)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.86)
- StackEval: Benchmarking LLMs in Coding Assistance (NeurIPS 2024 · similarity 0.86)
- SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.