Research map / Video / Video-language understanding
VideoGUI: A Benchmark for GUI Automation from Instructional Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VideoGUI, a hierarchical benchmark of 86 full tasks and 463 subtasks sourced from instructional videos for visual-centric GUI automation, evaluating high-level planning, middle-level planning, and atomic actions across LMMs and LLMs.
Problem
Existing GUI agent benchmarks focus on simple tasks described by a single text instruction, while real users struggle with novel, visual-centric, long-horizon software tasks such as video editing and visual creation.
Approach
Build tasks from instructional videos where annotators reproduce the demonstrated effects, recording human action trajectories, and evaluate agents hierarchically (visual-query high-level planning, visual+text middle-level action narration, and atomic click/drag/type/scroll execution) with per-dimension metrics.
Key results
- GPT-4o achieves the highest overall VideoGUI score of 39.4, with high-level planning only 17.1 on a 0-5 critic scale subset, indicating planning from visual queries is the bottleneck.
- Vision-only high-level planning scores are much lower than text-only (0.49 vs 2.37 averaged across models), showing visual-centric planning difficulty.
- GPT-4o with OCR reaches 56.3 action score, and GPT-4o with SoM reaches 54.3; the best full-task success rate of the GPT-4o agent on PowerPoint is 0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a visual-only query instead of a text query for high-level planning | VideoGUI models, average by models · VideoGUI high-level planning | planning score (0-5): 2.37 -> 0.49 | Hurt |
| Use a visual-only query instead of a text query for middle-level planning | VideoGUI models, average by models · VideoGUI middle-level planning | planning score (0-5): 2.02 -> 0.99 | Hurt |
| Add visual input to a text query | VideoGUI planning models · VideoGUI high- and middle-level planning | planning score (0-5): high-level: 2.37 -> 1.97; middle-level: 2.02 -> 1.98 | No clear effect |
| Add OCR | GPT-4o · VideoGUI | overall score (%): 39.4 -> 42.3 | Helped |
| Add OCR | GPT-4o · VideoGUI | click recall: 17.7 -> 60.1 | Helped |
| Add OCR | GPT-4o · VideoGUI | drag recall: 2.5 -> 11.3 (+8.8) | Helped |
| Add SoM | GPT-4o · VideoGUI | overall score (%): 39.4 -> 41.6 | Helped |
| Provide ground-truth high-level and middle-level plans (ablation) | GPT-4o GUI Agent · VideoGUI PowerPoint full tasks | Arena rank: 2.50 -> 1.38 | Helped |
| Provide a ground-truth middle-level plan (ablation) | GPT-4o GUI Agent · VideoGUI PowerPoint subtasks | success rate (%): 20.0 -> 50.0 | Helped |
| Provide a ground-truth middle-level plan (ablation) | GPT-4o GUI Agent · VideoGUI PowerPoint subtasks | average rounds: 5.4 -> 3.3 | Helped |
Surprising findings
- Even GPT-4o fails to complete a single full task end-to-end (0% success rate in the simulator), despite reasonable atomic action scores.
- Text-only GPT-3.5-Turbo with OCR achieves a higher action score (50.0) than several multimodal models, and planning from text does not gain from adding vision.
Limitations
- Full-task success is 0% for all agent variants, so end-to-end performance signal is limited.
- Planning evaluation relies on GPT-4-Turbo as a critic, which may introduce judge bias.
- Dataset is small (86 full tasks, 463 subtasks) and covers 11 applications.
Experimental setup
Datasets: VideoGUI
Benchmarks: VideoGUI
Metrics: normalized distance Recall@d precision accuracy LLM critic score (0-5) success rate arena rank
Baselines: GPT-4-Turbo Claude-3-Opus Gemini-Pro-Vision Qwen-VL-Max CogAgent LLama3-70B Mixtral-8x22B GPT-3.5-Turbo
Builds on: Set-of-Mark prompting chain-of-thought ReAct PyAutoGUI
Closest papers by meaning
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.90)
- GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding (ICLR 2025 · similarity 0.89)
- UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.89)
- Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining (ICML 2026 · similarity 0.89)
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.88)
- AssistGUI: Task-Oriented PC Graphical User Interface Automation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.