Research map / Multi-agent systems and games / LLM agents

GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents

CVPR 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces GUI-CEval, a Chinese mobile GUI agent benchmark on physical devices with 201 apps, a two-level (foundation QA + application grounding/offline/online) five-dimension structure, and evaluates 20 MLLMs/GUI models/multi-agent systems.

Problem

Existing GUI agent benchmarks are English-centric, span heterogeneous platforms, and focus on isolated skills (grounding or offline/online agent), lacking a unified fine-grained evaluation of Chinese mobile GUI agents from perception to execution.

Approach

A hierarchical benchmark pairing atomic multiple-choice QA across five dimensions (perception, planning, reflection, execution, evaluation) with application tasks (grounding, offline agent, online agent) built on the same apps and pages, with human-collected data on real Chinese mobile devices and multi-stage quality control.

Key results

• Qwen2.5-VL-72B achieves the highest average of 61.41%, followed by UI-TARS-72B-SFT (56.22%) and Qwen2.5-VL-32B (55.46%).
• GPT-4o and GPT-4o-mini reach only 27.69% and 20.93% average, well behind open-source models of similar scale.
• Best online agent success rate is 33.33% (UI-TARS-72B-SFT); Reflection and Evaluation remain weak (max 21.01% and 49.24%).

Surprising findings

• Closed-source GPT-4o and GPT-4o-mini trail open-source models of similar scale substantially.
• Online success drops sharply when the initial page deviates from the home screen (e.g., Qwen2.5-VL-3B from 27.7% to 0 in the Other condition).

Limitations

• Online evaluation is limited to apps that do not require login and to a fixed device setup, which may restrict coverage.
• Interpretive answers are drafted with GPT-4o before human verification, which may introduce model bias.
• Scaling gains are diminishing and the paper does not analyze all model configurations in the main table.

Experimental setup

Datasets: GUI-CEval

Benchmarks: GUI-CEval

Metrics: accuracy point-in-box accuracy step-wise accuracy task success rate step success rate

Baselines: GPT-4o GPT-4o-mini UI-TARS-7B-SFT UI-TARS-72B-SFT MIMO-VL-RL TongUI-7B OS-Atlas-base-7B Mobile-Agent v2

Builds on: ScreenSpot AndroidWorld Set-of-Marks

Closest papers by meaning

• VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics (ICML 2026 · similarity 0.91)
• Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigation (ICML 2026 · similarity 0.90)
• MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.90)
• GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding (ICLR 2025 · similarity 0.88)
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.88)
• UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.88)