Research map / Video / Video-language understanding
GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation
Rui Xie, Zhi Gao, Chenrui Shi, Zirui Shang, Lu Chen, Qing Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add planning knowledge from tutorial videos (ablation) | Seed-1.8 · OSWorld (361 tasks) | average score (%): 37.14 -> 43.93 | Improved |
| add grounding knowledge on top of planning (ablation) | Seed-1.8 · OSWorld (361 tasks) | average score (%): 43.93 -> 44.62 | Improved |
| add planning knowledge (ablation) | Qwen3-VL-8B · OSWorld (361 tasks) | average score (%): 33.90 -> 38.93 | Improved |
| add grounding on top of planning (ablation) | Qwen3-VL-8B · OSWorld (361 tasks) | average score (%): 38.93 -> 39.73 | Improved |
| add grounding knowledge only (no planning) (ablation) | Qwen3-VL-8B · OSWorld (361 tasks) | average score (%): 33.90 -> 34.31 | No clear effect |
| add GUIDE to multi-agent system | AgentS3 · OSWorld (361 tasks) | average score (%): 50.18 -> 54.65 | Improved |
| add GUIDE knowledge | Agents3+GPT-5.2 · WindowsAgentArena (154 tasks) | average score (%): 49.00 -> 59.21 | Improved |
| add GUIDE knowledge | Qwen3-VL-32B · WindowsAgentArena (154 tasks) | average score (%): 31.70 -> 44.16 | Improved |
| inject raw tutorial trajectories as ICL instead of structured knowledge (ablation) | Qwen3-VL-8B-thinking · OSWorld (361 tasks) | average score (%): 33.90 -> 31.96 | Hurt |
| add subtitle-derived planning knowledge only (ablation) | Qwen3-VL-8B-thinking · OSWorld balanced subset (120 tasks) | average score (%): 40.00 -> 42.32 | Improved |
| add visual pairwise annotation (without OmniParser) (ablation) | Qwen3-VL-8B-thinking · OSWorld balanced subset (120 tasks) | average score (%): 42.32 -> 47.52 | Improved |
| add OmniParser UI element graph (ablation) | Qwen3-VL-8B-thinking · OSWorld balanced subset (120 tasks) | average score (%): 47.52 -> 49.11 | Improved |
Closest papers by meaning
- Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining (ICML 2026 · similarity 0.89)
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.89)
- GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding (ICLR 2025 · similarity 0.88)
- AgentStudio: A Toolkit for Building General Virtual Agents (ICLR 2025 · similarity 0.87)
- VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks (ICLR 2025 · similarity 0.87)
- OS-ATLAS: Foundation Action Model for Generalist GUI Agents (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
