Research map / Multi-agent systems and games / LLM agents

Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining

ICML 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds Video2GUI, an automated coarse-to-fine pipeline that turns 500M YouTube metadata entries into WildGUI (12.7M GUI trajectories) and uses it for two-stage continual pretraining and SFT of Qwen2.5-VL-7B and Mimo-VL-7B.

Problem

GUI agents generalize poorly because large-scale, diverse GUI interaction trajectories are scarce and manual annotation is costly and domain-narrow.

Approach

Filter internet tutorial videos with a metadata classifier followed by a content-based omnimodal quality scorer, extract instruction-trajectory pairs with a VLM over sliding video segments, and ground actions to high-resolution screenshot triplets to yield pixel-accurate coordinates for pretraining.

Key results

• Mimo-VL-7B + WildGUI reaches 67.6 average on OSWorld-G, above Qwen3-VL-32B (60.6) and Seed1.5-VL (62.9).
• Qwen2.5-VL-7B + WildGUI improves ScreenSpot-Pro average from 26.8 to 41.9 (text/icon averages as reported) and Mimo-VL-7B reaches 56.9.
• On AndroidWorld, Mimo-VL-7B improves from 16.4 to 31.9 success rate with Stage 1 + Stage 2 training, versus 23.3 for Stage 2 only.

Surprising findings

• Removing the trajectory-modeling loss keeps static grounding and CAGUI performance competitive but drops AndroidWorld success sharply (31.9 to 24.1).
• Stage 2 post-training alone without Stage 1 pretraining collapses AndroidWorld success to 6.0.

Limitations

• Relies on proprietary Gemini-3-Pro for trajectory extraction and grounding annotation, so the pipeline is costly and not fully open.
• Grounding accuracy validated only on 200 manually checked actions and a 300-sample user study.
• Data is drawn only from YouTube, limiting source diversity.

Experimental setup

Datasets: WildGUI AndroidControl CAGUI open-source post-training GUI datasets

Benchmarks: ScreenSpot-Pro OSWorld-G AndroidControl CAGUI OSWorld AndroidWorld

Metrics: accuracy step success rate type accuracy average score

Baselines: Gemini-2.5-Pro Seed1.5-VL Qwen3-VL-32B UI-TARS-72B GTA1-7B UI-Venus-7B OpenCUA-7B GUI-Owl-7B OS-Genesis-7B OS-Atlas-7B Aguvis-7B UI-TARS-7B

Builds on: Aguvis TongUI VideoAgentTrek

Closest papers by meaning

• TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents (AAAI 2026 · similarity 0.92)
• ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.90)
• GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding (ICLR 2025 · similarity 0.89)
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.89)
• VideoGUI: A Benchmark for GUI Automation from Instructional Videos (NeurIPS 2024 · similarity 0.89)
• OS-ATLAS: Foundation Action Model for Generalist GUI Agents (ICLR 2025 · similarity 0.88)