Research map / Multi-agent systems and games / LLM agents
GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GUITAR, a state-centric framework that abstracts GUI screens into functional states to build a State Transition Graph, revealing failure concentration on bottleneck states and yielding a small gain from bottleneck-targeted guidance.
Problem
Step-level success rate treats each GUI screen independently, obscuring systematic failures across functionally equivalent screens and hiding failures on rare but critical states in long-tailed GUI trajectories.
Approach
Map visually diverse screens to shared functional states (by action-conditioned task role rather than visual similarity), aggregate trajectories into a global State Transition Graph annotated with state- and transition-level success rates, and use the identified bottleneck states to guide inference via retrieved sub-goals.
Key results
- 60.4% of failures occur in 20% of states (95% CI [36.1%, 76.2%]) across 8 agents x 6 tasks.
- Bottleneck-targeted guidance improves SR by +2.8% on average, versus -3.2% for uniform guidance.
- Under three-fold trajectory-held-out evaluation with automatic STGs, guidance gives +1.88% mean SR gain across 36 task-agent pairs for 7 agents.
Surprising findings
- Uniform guidance applied to all states lowers average SR (-3.2%), while targeted guidance raises it.
- Gains vary substantially across agents and tasks; several task-agent pairs decrease under held-out evaluation.
Limitations
- STG quality depends on automatic state assignment; manual correction rates range 0-25.3% and assignment errors can shift bottlenecks.
- Evaluated only within the same apps/websites; no transfer to unseen applications or domains.
- STGs are built offline from a fixed set of reference trajectories, so rare unseen states may be missing.
Experimental setup
Datasets: AndroidControl Mind2Web
Benchmarks: AndroidControl Mind2Web
Metrics: step success rate state success rate transition success rate Gini coefficient Delta SR Jaccard Spearman rho Gwet's AC1
Baselines: Step-level SR PageAgent ImageHash WebGraphEval Uniform guidance
Builds on: PageAgent
Closest papers by meaning
- ProBench: Benchmarking GUI Agents with Accurate Process Information (AAAI 2026 · similarity 0.89)
- DiagEval: Trajectory-Conditioned Diagnosis for Reliable Software Evaluation with GUI Agents (NeurIPS 2026 · similarity 0.88)
- Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents (ICML 2026 · similarity 0.87)
- D-GARA: A Dynamic Benchmarking Framework for GUI Agent Robustness in Real-World Anomalies (AAAI 2026 · similarity 0.86)
- Agent-SAMA: State-Aware Mobile Assistant (AAAI 2026 · similarity 0.86)
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.