Research map / Multi-agent systems and games / LLM agents
GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GUI-Xplore, a benchmark of app exploration videos with five multiple-choice downstream tasks, and Xplore-Agent, a baseline that converts exploration videos into a GUI transition graph for LLM reasoning.
Problem
GUI agents generalize poorly across unfamiliar apps and tasks because existing datasets ignore developer-induced structural variation across apps and focus mostly on navigation tasks.
Approach
Provide agents with pre-recorded app exploration videos as priors, convert them into action-aware keyframes, view hierarchies, and a GUI transition graph, then answer five hierarchical multiple-choice tasks with an LLM.
Key results
- Xplore-Agent reaches 30.73% element accuracy, 84.63% operation accuracy and 30.39% step success rate on the cross-app automation test set, vs. 17.18%/73.54%/15.80% for the best baseline CogAgent.
- On GUI-Xplore's five tasks, Xplore-Agent averages 64.24% accuracy, above the best end-to-end model GPT (59.39% at 8 frames) and VideoTree (57.97%).
- Rule-based GUI clustering yields 61.87% average accuracy at 63,771 tokens per app, while GPT-based clustering yields 64.24% at 45,199 tokens per app.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| provide exploration-video priors and use Xplore-Agent | GUI agents · Cross-App Task Automation test set | Ele. Acc.: 17.18% -> 30.73% | Helped |
| provide exploration-video priors and use Xplore-Agent | GUI agents · Cross-App Task Automation test set | Op. Acc.: 73.54% -> 84.63% | Helped |
| provide exploration-video priors and use Xplore-Agent | GUI agents · Cross-App Task Automation test set | StepSR: 15.80% -> 30.39% | Helped |
| use Xplore-Agent with action-aware GUI modeling and graph-guided reasoning | GUI-Xplore · Application Overview | accuracy: 96.88% -> 99.25% | Helped |
| use Xplore-Agent with action-aware GUI modeling and graph-guided reasoning | GUI-Xplore · Page Analysis | accuracy: 82.12% -> 92.86% | Helped |
| use Xplore-Agent with action-aware GUI modeling and graph-guided reasoning | GUI-Xplore · Application Usage | accuracy: 67.04% -> 68.21% | Helped |
| use Xplore-Agent with action-aware GUI modeling and graph-guided reasoning | GUI-Xplore · Action Recall | accuracy: 24.09% -> 24.36% | Helped |
| use Xplore-Agent with action-aware GUI modeling and graph-guided reasoning | GUI-Xplore · Action Sequence Verification | accuracy: 28.85% -> 36.54% | Helped |
| replace rule-based GUI clustering with GPT clustering (ablation) | Xplore-Agent · GUI-Xplore | Avg.: 61.87% -> 64.24% | Helped |
| replace rule-based GUI clustering with GPT clustering (ablation) | Xplore-Agent · GUI-Xplore | Avg. Token /App: 63771 -> 45199 | Helped |
| replace 1 FPS sampling with action-aware keyframe extraction (ablation) | Xplore-Agent · GUI-Xplore | Num./app: 629.35 -> 115.75 | Helped |
| replace 1 FPS sampling with action-aware keyframe extraction (ablation) | Xplore-Agent · GUI-Xplore | Page Num./100 frames: 54.95 -> 10.11 | Hurt |
Surprising findings
- Increasing the number of input frames often degraded end-to-end video model performance on GUI-Xplore.
- All models performed better on Environment Understanding than on Operational Behavior Understanding tasks.
Limitations
- Models only output textual answers rather than concrete GUI actions, so the work does not yet yield a fully generalized agent.
- Exploration video collection faces data source and privacy concerns.
- Cross-app test set is small (20 apps, 500 operations) and the comparison with GUI agent baselines is not fully matched in task format.
Experimental setup
Datasets: GUI-Xplore Mind2Web-style cross-app automation test set derived from GUI-Xplore
Benchmarks: GUI-Xplore Cross-App Automation test set (500 operations, 20 apps)
Metrics: Element Accuracy Operation Accuracy StepSR multiple-choice accuracy
Baselines: GPT-4o CogVLM2-Video VideoChat2 VideoLLaMA2 VideoTree CogAgent SeeClick AUTO-UI
Builds on: Pix2Struct VideoTree DroidBot AutoDroid
Closest papers by meaning
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.90)
- TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents (AAAI 2026 · similarity 0.90)
- GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.89)
- ProBench: Benchmarking GUI Agents with Accurate Process Information (AAAI 2026 · similarity 0.89)
- HATS: Hardness-Aware Trajectory Synthesis for GUI Agents (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.