Research map / Multi-agent systems and games / LLM agents

Watch and Learn: Learning to Use Computers from Online Videos

CVPR 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Watch & Learn converts 53K YouTube tutorial videos into UI trajectories using a vision-only inverse dynamics model trained on 600K web transitions, then uses them as in-context exemplars and SFT data for computer-using agents.

Problem

Computer-using agents lack large-scale, high-quality human demonstration data; existing synthetic or MLLM-labeled trajectory pipelines are noisy, narrow, or costly.

Approach

Frame trajectory recovery from videos as inverse dynamics: a small SigLIP-2 + Transformer model predicts the action between consecutive screenshots, trained on automatically logged web transitions, and applied to retrieved tutorial videos to label them without manual annotation or MLLM labeling.

Key results

• Qwen2.5-VL-7B SFT on IDM-labeled W&L trajectories rises from 1.9% to 13.0% on OSWorld-Verified (+11.1), versus 5.4% with TongUI labels.
• IDM labeling reaches 95.8% ActionType accuracy vs 84.3% for TongUI and 81.5% for Gemini 2.5 Flash on a held-out transition test set.
• UI-TARS-1.5-7B fine-tuned on W&L data reaches 24.0% on WindowsAgentArena (15-step), up from 18.1% zero-shot, while TongUI-labeled SFT drops to 12.9%.

Surprising findings

• Random video retrieval gives no gain over the o3 base model (21.8%), suggesting exemplar gains depend on task-relevant retrieval.
• SFT on TongUI-labeled videos degrades UI-TARS-1.5-7B on WindowsAgentArena (-5.2) and OSWorld (-3.5), indicating noisy labels can hurt.

Limitations

• Does not handle keyboard shortcut or hotkey actions since they are absent from web transition data.
• Retrieval operates at full-task granularity, which may not match agent sub-step needs.
• Random-retrieval ablation and gains are small for general models (about +1.6 to +3.0 points).

Experimental setup

Datasets: Common Crawl-based state-transition corpus (600K) YouTube tutorial videos (53,125) OSWorld-Verified WindowsAgentArena

Benchmarks: OSWorld-Verified WindowsAgentArena

Metrics: success rate ActionType accuracy action accuracy

Baselines: TongUI Gemini 2.5 Flash labeling Jedi OpenCUA-7B UltraCUA-7B UI-TARS-1.5-7B zero-shot

Builds on: VPT WebDreamer TongUI SigLIP-2 OSWorld

Closest papers by meaning

• Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining (ICML 2026 · similarity 0.88)
• AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials (ICLR 2025 · similarity 0.87)
• VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks (ICLR 2025 · similarity 0.87)
• ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.87)
• Grounding Video Models to Actions through Goal Conditioned Exploration (ICLR 2025 · similarity 0.87)
• WebChain: A Large-Scale Human-Annotated Dataset of Real-World Web Interaction Traces (CVPR 2026 · similarity 0.86)