Research map / Multi-agent systems and games / LLM agents

Programming with Pixels: Can Computer-Use Agents do Software Engineering?

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Programming with Pixels (PwP), a VSCode-based computer-use environment, and PwP-Bench (15 SWE tasks, 5400 instances; Lite 300) to test whether computer-use agents can match specialized SWE agents.

Problem

Unclear whether generalist computer-use agents that act via screenshots and keyboard/mouse can automate sophisticated software engineering work that specialized SWE agents handle with hand-engineered APIs.

Approach

A general SWE environment where agents visually control a VSCode IDE, plus a unified benchmark porting 13 existing SWE datasets and 2 new task sets; compares pure visual CUAs, CUAs with file/bash text APIs, and a specialized agent (mini-SWE-agent).

Key results

• Pure visual computer-use agents reach at most 22.9% overall average (Claude-Sonnet-4.0), well below mini-SWE-agent at 48.8%.
• Adding file-edit and bash text APIs raises Claude-Sonnet-4.0 to 50.7% overall average, matching or exceeding mini-SWE-agent on some categories.
• Claude-Sonnet-4.0 pure CUA has visual grounding errors in 95% of trajectories versus 20% for GPT-4o.

Surprising findings

• Set-of-Marks did not resolve grounding issues; agents selected incorrect elements instead.
• Claude-Sonnet-4.0 with explicit instruction to use rename/move tools improved refactoring accuracy from 25% to 75%, showing latent tool-use ability unused by default.

Limitations

• Evaluation on a 300-instance Lite subset of 20 per task due to cost, limiting comparability to specialized agent scores.
• Specialized-agent comparisons draw on heterogeneous published numbers with differing setups.
• Max 20 steps per task for main experiments.

Experimental setup

Datasets: HumanEval SWE-bench SWE-Bench-Multilingual DSBench ResQ CanItEdit Design2Code ChartMimic SWT-Bench Bird Intercode-CTF Minictx

Benchmarks: PwP-Bench PwP-Bench-Lite

Metrics: accuracy macro average of task-specific metrics

Baselines: mini-SWE-agent Claude-Sonnet-4.0 computer-use GPT-4o Gemini-Pro-1.5 Qwen2.5-VL-72B GUI-Owl-32B

Builds on: Set-of-Marks OSWorld mini-swe-agent Anthropic computer use

Closest papers by meaning

• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (NeurIPS 2024 · similarity 0.88)
• SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? (ICLR 2025 · similarity 0.87)
• ProgramBench: Can Language Models Rebuild Programs From Scratch? (NeurIPS 2026 · similarity 0.87)
• TOM-SWE: User Mental Modeling For Software Engineering Agents (ICML 2026 · similarity 0.87)
• SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents (NeurIPS 2025 · similarity 0.86)
• OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (ICLR 2026 · similarity 0.86)