Research map / Multi-agent systems and games / LLM agents

MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents

CVPR 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MMBench-GUI, a four-level hierarchical multi-platform benchmark (content understanding, element grounding, task automation, task collaboration) with an Efficiency-Quality-Aware (EQA) metric, and benchmarks open and proprietary VLMs/GUI agents.

Problem

Existing GUI agent benchmarks test isolated skills, focus on success rate while ignoring efficiency, and cover narrow sets of platforms (e.g., lacking macOS), making systematic cross-level diagnosis hard.

Approach

A unified hierarchical benchmark spanning perception, grounding, single-app automation, and multi-app collaboration over six platforms, plus EQA, a step-budget-aware score that approximates the area under the success-vs-step-budget curve to penalize redundant actions.

Key results

• InternVL3-72B achieves the top L1 overall scores (79.15% Easy, 77.89% Medium, 75.70% Hard) with the smallest drop across difficulty.
• UI-TARS-72B-DPO reaches 74.25% average L2 grounding accuracy, while GPT-4o and Claude-3.7 score under 5%.
• The best L3 system (GPT-4o + UI-TARS-1.5-7B) reaches 26.6% SR at 15 steps, while the best L4 system reaches only 8.78% SR.

Surprising findings

• General-purpose models such as GPT-4o and Claude-3.7 score near zero on L2 grounding, yet combining GPT-4o with a grounder (UGround or UI-TARS) raises L3 SR from 6.13% to over 17%.
• Multi-application L4 performance is very low, with the top system reaching only 8.78% SR.

Limitations

• L1 and L2 items are partly generated by proprietary models (Claude 3.7, GPT-o4-mini, GPT-o3), which may introduce model-specific biases despite human review.
• L3/L4 tasks rely on virtualized environments and tasks with unstable or account-dependent behavior were removed, limiting realism.
• MacOSArena contains only 70 tasks, a small sample for macOS evaluation.

Experimental setup

Datasets: ScreenSpot OSWorld AndroidWorld WebArena-Lite-v2 WindowsAgentArena MacOSArena

Benchmarks: MMBench-GUI

Metrics: accuracy point-in-box accuracy success rate EQA

Baselines: GPT-4o Claude-3.7 Qwen-Max-VL Qwen2.5-VL-72B UI-TARS-72B-DPO InternVL3-72B UGround-V1-7B Aguvis-72B ShowUI-2B OS-Atlas-Base-7B

Builds on: ScreenSpot OSWorld AndroidWorld WebArena-Lite WindowsAgentArena

Closest papers by meaning

• ProBench: Benchmarking GUI Agents with Accurate Process Information (AAAI 2026 · similarity 0.93)
• VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics (ICML 2026 · similarity 0.92)
• OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (ICLR 2026 · similarity 0.91)
• AssistGUI: Task-Oriented PC Graphical User Interface Automation (CVPR 2024 · similarity 0.91)
• UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.91)
• Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.90)