Research map / Multi-agent systems and games / LLM agents

DocOS: Towards Proactive Document-Guided Actions in GUI Agents

ICML 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the Proactive Document-Guided Action paradigm and the DocOS benchmark (817 desktop tasks, 20 apps) where GUI agents must search the web for official docs and then execute tasks, showing that current agents are limited by poor document localization and poor grounding of retrieved instructions.

Problem

GUI agents rely on static parametric knowledge, which fails on long-tailed, application-specific tasks needing explicit procedural documentation; no existing benchmark evaluates autonomous document search followed by document-grounded execution.

Approach

A paradigm where agents first proactively browse the open web to find official documentation (retrieval phase) and then ground its procedural instructions into GUI actions (execution phase), evaluated end-to-end in a Docker/QEMU desktop environment with URL-based navigation metrics (TUI, HPP) and task completion rate (TCR).

Key results

• Best agent UI-TARS-1.5-7B reaches 17.31% average TCR on DocOS, and TCR falls as task difficulty rises across agents.
• Qwen3-VL-8B achieves about 42% TUI on Easy tasks but only about 4% average TCR, indicating localization fails to translate into completion.
• Providing oracle documents raises TCR only modestly for most models (e.g., UI-TARS-1.5-7B 17.31 to 17.98), while MAI-UI-8B improves from 10.96 to 12.04.

Surprising findings

• Strong models such as UI-TARS-1.5-7B and Qwen3-VL-8B gain little from oracle documents, suggesting a context-processing bottleneck rather than missing information.
• Larger Qwen3-VL-32B does not substantially outperform the 8B model (TCR 6.21 vs 4.23).

Limitations

• Overall task completion rates are very low, so the benchmark's signal depends on small success counts.
• Execution-phase evaluation relies on an LLM judge with a rubric of ground-truth steps, which may be imprecise.
• Results are limited to the specific agents and 20 desktop applications evaluated.

Experimental setup

Datasets: DocOS

Benchmarks: DocOS

Metrics: TUI HPP TCR

Baselines: Qwen3-VL-8B UI-TARS-1.5-7B MAI-UI-8B GELab-Zero-4B-preview Aguvis-7B GUI-Owl-7B Qwen3-VL-32B AutoGen

Builds on: OSWorld

Closest papers by meaning

• Agent S: An Open Agentic Framework that Uses Computers Like a Human (ICLR 2025 · similarity 0.90)
• MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.89)
• UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.88)
• GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.88)