Research map / Multi-agent systems and games / LLM agents

ProSoftArena: Benchmarking Hierarchical Capabilities of Multi-modal Agents in Professional Software Environments

CVPR 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ProSoftArena, a 436-task benchmark and VM-based platform for evaluating multimodal computer-use agents on 13 professional software applications, with a five-level capability hierarchy, execution-based evaluation, and human-in-the-loop modes.

Problem

Existing computer-use agent benchmarks focus on browsers and general desktop apps, leaving professional scientific and industrial software workflows unevaluated.

Approach

A hierarchical capability taxonomy (L1 GUI operation to L5 project orchestration) for professional software, instantiated as an executable Windows VM benchmark with script-based execution checks and two human-in-the-loop modes (human takeover and agent-initiated assistance).

Key results

• Best agent Claude 4 Computer Use reaches 24.4% overall success rate on L2 tasks and 0.0% on L3 tasks.
• GPT-5 with a11y tree achieves 45.1% overall L1 success rate and 13.5% L2 success rate.
• With human-initiated takeover, Qwen2.5-VL's L2 success rises from 6.7% to 66.7% on VSCode tasks while steps drop from 44.6 to 12.5.

Surprising findings

• Agent-initiated assistance gives limited gains because agents rarely request help under uncertainty.
• Adding both screenshot and a11y tree on top of SoM reduced performance for Qwen2.5-VL on L1 tasks.

Limitations

• L3 and L4 subsets are very small (10 tasks each), and L5 is not covered.
• L4 creative tasks are evaluated subjectively rather than by execution-based metrics.
• Human-in-the-loop results are reported on only one application (VSCode) with two models.

Experimental setup

Benchmarks: ProSoftArena

Metrics: success rate steps human intervention count human operation time

Baselines: GPT-4o o3 GPT-5 GLM-4.5V Qwen2.5-VL Claude 4 Computer Use

Builds on: OSWorld Set-of-Mark prompting OmniParser

Closest papers by meaning

• Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.90)
• WorkArena: How Capable are Web Agents at Solving Common Knowledge Work Tasks? (ICML 2024 · similarity 0.89)
• UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.89)
• OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (ICLR 2026 · similarity 0.88)
• SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows? (NeurIPS 2026 · similarity 0.87)
• WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks (NeurIPS 2024 · similarity 0.87)