Research map / Multi-agent systems and games / LLM agents

TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents

AAAI 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Converts about 1M multimodal web tutorials (YouTube, Bilibili, WikiHow, Baidu Experience) into GUI agent trajectories (GUI-Net-1M) and fine-tunes Qwen2.5-VL-3B/7B/32B into TongUI agents that improve grounding and navigation benchmarks.

Problem

Generalized GUI agents lack large-scale, diverse trajectory data across operating systems and applications, since manual annotation is expensive and synthetic trajectories may lack diversity and accuracy.

Approach

Mine readily available multimodal web tutorials (videos with ASR transcripts and image-text articles) and convert them into step-level screenshot-thought-action trajectories using an LLM-based text/visual pipeline, a zero-shot UI-TARS agent for action labeling, and multi-stage filtering, with no simulators.

Key results

• TongUI-7B reaches 86.0 on ScreenSpot and 88.7 on ScreenSpot-V2 average, versus 78.6 and 84.0 for the reproduced Qwen2.5-VL-7B baseline.
• TongUI-32B scores 33.1 average on ScreenSpot-Pro, compared with UI-TARS-7B at 35.7 and UI-TARS-72B at 38.1.
• TongUI-7B achieves 73.9 on MiniWob versus 71.5 for ShowUI-2B; TongUI-32B reaches 74.3.

Surprising findings

• Filtering raised the average human quality score of sampled trajectories from 3.22 to 4.12, and GUI-Net-1M is rated comparable to ShowUI data (4.26).

Limitations

• Trajectory actions are labeled by a zero-shot UI-TARS agent, so label noise is inherited from the teacher model.
• Quality is assessed by a small user study of 100-sample sets rated by five participants.
• Tutorial-derived trajectories are offline and the method is not iterated or updated continually.

Experimental setup

Datasets: GUI-Net-1M ShowUI AndroidControl training set GUI Odyssey training set Mind2Web training data

Benchmarks: ScreenSpot ScreenSpot-V2 ScreenSpot-Pro AITW AndroidControl Mind2Web UI-Vision MiniWob GUI Odyssey Baidu Experience

Metrics: accuracy step accuracy step success rate score element accuracy operation F1

Baselines: ShowUI-2B UI-TARS-2B UI-TARS-7B UI-TARS-72B OS-Atlas-7B AGUVIS-7B SeeClick-9.6B Qwen2.5-VL-3B Qwen2.5-VL-7B

Builds on: ReAct Whisper UI-TARS Qwen2.5-VL MOG2

Closest papers by meaning

• Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining (ICML 2026 · similarity 0.92)
• GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration (CVPR 2025 · similarity 0.90)
• OS-ATLAS: Foundation Action Model for Generalist GUI Agents (ICLR 2025 · similarity 0.89)
• TreeCUA: Efficiently Scaling GUI Automation with Tree-Structured Verifiable Evolution (ICML 2026 · similarity 0.89)
• ProBench: Benchmarking GUI Agents with Accurate Process Information (AAAI 2026 · similarity 0.89)
• MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI Agents (CVPR 2026 · similarity 0.89)