Research map / Multi-agent systems and games / LLM agents
SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SWE-bench Multimodal, a 617/619-instance benchmark of visual JavaScript GitHub issues, and shows that existing SWE-bench systems adapted to it resolve at most 12.2% of tasks.
Problem
Existing repository-level software engineering benchmarks such as SWE-bench are Python-only and mostly text-only, so it is unknown whether AI systems generalize to visual, user-facing software in other languages.
Approach
Builds a SWE-bench-style benchmark from 17 user-facing JavaScript/TypeScript repositories, keeping only task instances whose issues or unit tests include images or videos, and adapts existing systems (SWE-agent, Agentless, RAG) to JavaScript and multimodal inputs to measure generalization.
Key results
- SWE-agent M with GPT-4o resolves 12.2% of SWE-bench M test instances, the best system; Claude 3.5 Sonnet SWE-agent Base also reaches 12.2%.
- Agentless JS reaches only 3.1-6.2% resolved and RAG 5.0-6.0%, showing Python-specific localization fails to transfer.
- Removing images lowers RAG with Claude 3.5 Sonnet from 14.1% to 11.2% and SWE-agent JS with Claude from 16.0% to 13.0% on the development split.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add problem statement images (ablation) | SWE-agent JS with GPT-4o · SWE-bench M development split | % Resolved: 8.0 -> 11.0 | Helped |
| Add problem statement images (ablation) | SWE-agent JS with Claude 3.5 Sonnet · SWE-bench M development split | % Resolved: 13.0 -> 16.0 | Helped |
| Add problem statement images (ablation) | RAG with GPT-4o · SWE-bench M development split | % Resolved: 8.0 -> 10.0 | Helped |
| Add problem statement images (ablation) | RAG with Claude 3.5 Sonnet · SWE-bench M development split | % Resolved: 11.2 -> 14.1 | Helped |
| Add SWE-agent's JavaScript-specific editing interface (ablation) | SWE-agent · SWE-bench M | % Resolved | No clear effect |
| Add SWE-agent's browser, screenshot, and image-viewing tools (ablation) | SWE-agent with GPT-4o · SWE-bench M development split | fraction of correct submissions: 10.4% -> 19.6% | Helped |
| Add SWE-agent's browser, screenshot, and image-viewing tools (ablation) | SWE-agent with Claude 3.5 Sonnet · SWE-bench M development split | % Resolved | Hurt |
| Use a custom JavaScript parser compatible with Agentless instead of the Python AST module | Agentless · SWE-bench M development split | % Resolved: 0% -> 4.6% | Helped |
| Provide images containing non-textual visual information (ablation) | SWE-agent JS · SWE-bench M development split, non-textual visuals | % Resolved: 8.7% -> 13.0% | Helped |
Surprising findings
- Adding multimodal web tools to SWE-agent degraded performance with Claude 3.5 Sonnet, while GPT-4o improved, and the gains were ambiguous overall.
- Agentless JS, after replacing Python ast with a custom JavaScript parser, still performed poorly (0% on the dev split before adaptation), indicating Python-centric localization assumptions.
Limitations
- Only JavaScript/TypeScript repositories with user-facing visual content; generalization to other languages or domains is not tested.
- Only two proprietary multimodal LMs (GPT-4o and Claude 3.5 Sonnet) evaluated; open models excluded for context and multimodal constraints.
- Several existing systems (AutoCodeRover, Moatless) were not benchmarked because adaptation was infeasible.
Experimental setup
Datasets: SWE-bench
Benchmarks: SWE-bench Multimodal
Metrics: % Resolved Avg. $ Cost
Baselines: SWE-agent Base SWE-agent JS SWE-agent M Agentless JS RAG
Builds on: SWE-bench SWE-agent Agentless BM25
Closest papers by meaning
- SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents (NeurIPS 2025 · similarity 0.88)
- SWE-bench Goes Live! (NeurIPS 2025 · similarity 0.88)
- SWE-bench: Can Language Models Resolve Real-world Github Issues? (ICLR 2024 · similarity 0.87)
- Programming with Pixels: Can Computer-Use Agents do Software Engineering? (ICLR 2026 · similarity 0.87)
- SWE-smith: Scaling Data for Software Engineering Agents (NeurIPS 2025 · similarity 0.87)
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.