Research map / Multi-agent systems and games / LLM agents

ProBench: Benchmarking GUI Agents with Accurate Process Information

AAAI 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ProBench, a bilingual mobile GUI benchmark of 200+ tasks split into state-related and process-related types, with an automated Process Provider (accessibility-tree converter and MLLM summarizer) that records intermediate actions for evaluation.

Problem

Existing GUI agent benchmarks judge task completion only from the final screen, missing critical intermediate operation steps, and prior process-aware evaluations rely on manual annotation or LLM decomposition that is not accurate or scalable.

Approach

Separate State-related tasks (judged by final screenshot) from Process-related tasks (judged by final screen plus automatically captured process information), where process info is produced by an accessibility-tree-based Structure Description Converter or an MLLM-based Summarizer comparing before/after screenshots.

Key results

• Gemini 2.5 Pro achieves the highest overall average accuracy of 40.1% (ST 45.6%, PT 27.9%) on ProBench.
• Qwen2.5-VL-72B reaches 53.3% English average, the best English score among open models, while GPT-4o and Claude 4 Sonnet score 0.0%.
• The evaluator with Structure Description Converter reaches 89.7% correctness on process-related tasks, and with MLLM Summarizer 94.1%, versus 96.0% for state-related tasks.

Surprising findings

• GPT-4o and Claude 4 Sonnet score 0% overall, attributed to poor grounding of click coordinates.
• UI-R1-E-3B fails every task because it never emits the COMPLETE action seen in its training.

Limitations

• Evaluation relies on binary success judgments rather than degrees of task progress.
• Benchmark is small (over 200 tasks across 34 apps) and only 15 steps per task.
• Process-related tasks have limited accuracy with the Structure Description Converter (89.7%).

Experimental setup

Datasets: ProBench

Benchmarks: ProBench

Metrics: accuracy (success rate) early stop ratio evaluator correctness vs human annotation

Baselines: GPT-4o Claude 4 Sonnet Gemini 2.5 Pro Qwen2.5-VL-7B Qwen2.5-VL-32B Qwen2.5-VL-72B InternVL3-8B UI-TARS-1.5-7B UI-R1-E-3B GUI-R1-3B

Builds on: SPA-Bench AITW AndroidWorld A3

Closest papers by meaning

• MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.93)
• VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics (ICML 2026 · similarity 0.90)
• TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents (AAAI 2026 · similarity 0.89)
• AssistGUI: Task-Oriented PC Graphical User Interface Automation (CVPR 2024 · similarity 0.89)
• OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (ICLR 2026 · similarity 0.89)
• HATS: Hardness-Aware Trajectory Synthesis for GUI Agents (CVPR 2026 · similarity 0.89)