Research map / Multi-agent systems and games / LLM agents
AssistGUI: Task-Oriented PC Graphical User Interface Automation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AssistGUI, a 100-task benchmark for desktop (Windows) GUI automation with productivity software, and an Actor-Critic embodied agent with an LLM-driven GUI parser and hierarchical planner, reaching 46% success with GPT-4.
Problem
Existing GUI automation benchmarks focus on Android, web, and terminal tasks that are entertainment-oriented; there is no benchmark for task-oriented desktop productivity software that requires dense GUI understanding, complex operations, and long procedures.
Approach
Combine an instructional video and user query to plan milestones/subtasks, then iterate GUI parsing (metadata, OCR, icon matching, detectors, segmentation), a Critic that judges previous actions from before/after screenshots, and an Actor that generates PyAutoGUI code, with outcome-based evaluation of project files.
Key results
- The full Actor-Critic agent with GPT-4 achieves 46.0% overall success on the 100-task benchmark, versus 32.0% for ReAct and 12.0% for CoT.
- Removing OCR drops success to 4.0% and removing icon parsing drops it to 13.0%, showing GUI parsing components are critical.
- Human non-expert participants who watched the video once reach 85.0% overall success, far above the best model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AssistGUI's hierarchical planning and Actor-Critic framework instead of CoT | AssistGUI · AssistGUI overall | success rate: 12.0 -> 46.0 | Helped |
| use AssistGUI's hierarchical planning and Actor-Critic framework instead of ReAct | AssistGUI · AssistGUI overall | success rate: 32.0 -> 46.0 | Helped |
| add planning; parse the video subtitle into subtasks rather than feed the full subtitle to the Actor (ablation) | AssistGUI · AssistGUI overall | success rate: 35.0 -> 46.0 | Helped |
| add the Critic (ablation) | AssistGUI · AssistGUI overall | success rate: 41.0 -> 46.0 | Helped |
| include instructional-video subtitles in planning (ablation) | AssistGUI · AssistGUI overall | success rate: 37.0 -> 46.0 | Helped |
| include panel layout in the parsed GUI (ablation) | AssistGUI · AssistGUI overall | success rate: 44.0 -> 46.0 | Helped |
| include icons in the parsed GUI (ablation) | AssistGUI · AssistGUI overall | success rate: 13.0 -> 46.0 | Helped |
| include OCR in the parsed GUI (ablation) | AssistGUI · AssistGUI overall | success rate: 4.0 -> 46.0 | Helped |
| include other parsed GUI elements, such as footage content and scrolls (ablation) | AssistGUI · AssistGUI overall | success rate: 43.0 -> 46.0 | Helped |
| replace the GUI Parser with Qwen-VL-Chat (ablation) | AssistGUI · AssistGUI overall | success rate: 46.0 -> 5.0 | Hurt |
| replace GPT-4 with GPT-3.5 for the Actor and Critic (ablation) | AssistGUI · AssistGUI overall | success rate: 46.0 -> 12.0 | Hurt |
| replace GPT-4 with Llama2 for the Actor and Critic (ablation) | AssistGUI · AssistGUI overall | success rate: 46.0 -> 1.0 | Hurt |
Surprising findings
- GPT-4V-SoM performance is nearly nullified on desktop GUIs due to Semantic-SAM segmentation limits.
- Removing panel layout information had minimal impact (44.0% vs 46.0%), suggesting GPT-4 recovers buttons without it.
Limitations
- Complex footage operations such as roto brush selection are not supported by the agent.
- Critic judgments are unreliable for complex scenarios requiring fine action-vision alignment.
- Open-source models such as Llama2-7B and GPT-3.5 perform poorly without fine-tuning.
Experimental setup
Datasets: instructional videos from official software websites and video-sharing platforms official and annotator-created project files
Benchmarks: AssistGUI
Metrics: success rate pixel-similarity threshold for Design/Office final-screenshot match for Widget script-checked settings and folder structure
Baselines: CoT ReAct GPT-4V-SoM Qwen-VL-Chat GPT-3.5 Llama2-7B
Builds on: ReAct Chain-of-thought MMReAct VisualClues GPT-4V-SoM
Closest papers by meaning
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.91)
- Agent S: An Open Agentic Framework that Uses Computers Like a Human (ICLR 2025 · similarity 0.90)
- ProBench: Benchmarking GUI Agents with Accurate Process Information (AAAI 2026 · similarity 0.89)
- UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.88)
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.88)
- VideoGUI: A Benchmark for GUI Automation from Instructional Videos (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.