Research map / Large language models / AI and society
GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks
Saelyne Yang, Jaesang Yu, Yi-Hao Peng, Kevin Qinghong Lin, Jae Won Cho, Yale Song, Juho Kim
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| provide behavior state as context for help need detection (ablation) | GPT-4o · GUIDE Help Need Detection | F1: 47.73 -> 90.19 | Improved |
| provide behavior state as context for help need detection (ablation) | Claude-4.5-Sonnet · GUIDE Help Need Detection | accuracy: 39.49 -> 58.56 | Improved |
| add behavior state and intent to help content prediction (ablation) | InternVideo2.5-8B · GUIDE Help Content Prediction | accuracy: 23.67 -> 73.86 | Improved |
| add behavior state and intent to help content prediction (ablation) | Claude-4.5-Sonnet · GUIDE Help Content Prediction | accuracy: 55.00 -> 82.79 | Improved |
| add intent on top of behavior state for help need detection (ablation) | Gemini-2.5-Pro · GUIDE Help Need Detection | accuracy: 84.73 -> 82.38 | Hurt |
| add intent on top of behavior state for help need detection (ablation) | GPT-4o · GUIDE Help Need Detection | accuracy: 87.79 -> 87.91 | No clear effect |
| add previous behavior state for behavior detection (ablation) | MLLMs · GUIDE Behavior State Detection | accuracy: 21.57 -> 27.02 (InternVideo2.5-8B) | Improved |
| add behavior state context for intent prediction (ablation) | MLLMs · GUIDE Intent Prediction | accuracy: 71.39 -> 72.62 (Claude-4.5-Sonnet) | Improved |
| give models progressively more of the video segment (25% to 100%) (ablation) | MLLMs · GUIDE online setting | accuracy | Improved |
| use stricter MBAcc instead of accuracy | MLLMs · GUIDE Intent Prediction | metric: 71.39 -> 65.44 (Claude-4.5-Sonnet) | Hurt |
Closest papers by meaning
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.91)
- UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.90)
- AssistGUI: Task-Oriented PC Graphical User Interface Automation (CVPR 2024 · similarity 0.90)
- VideoGUI: A Benchmark for GUI Automation from Instructional Videos (NeurIPS 2024 · similarity 0.89)
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.89)
- ProBench: Benchmarking GUI Agents with Accurate Process Information (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
