Research map / Video / Action recognition
VideoAgentTrek: Computer-Use Pretraining from Unlabeled Videos
Dunjie Lu, Yiheng Xu, Junli Wang, Haoyuan Wu, Xinyuan Wang, Zekun Wang, Junlin Yang, Hongjin SU, Jixuan Chen, Junda Chen, Yuchen Mao, Junyang Lin and 2 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add stage 1 continued pretraining on mined video trajectories before SFT (ablation) | Qwen2.5-VL-7B · OSWorld-Verified | task success rate (%): 9.3 -> 14.13 (20 steps); 9.27 -> 15.78 (50 steps) | Improved |
| add stage 1 continued pretraining on mined video trajectories before SFT (ablation) | Qwen2.5-VL-7B · AgentNetBench | step success rate: 64.1% -> 69.3% | Improved |
| scale video pretraining tokens from 0% to 50% to 100% (ablation) | VideoAgentTrek · AgentNetBench | step success rate: 64.1% -> 68.1% -> 69.3% | Improved |
| scale video pretraining tokens from 0% to 50% to 100% (ablation) | VideoAgentTrek · OSWorld-Verified SR@50 | task success rate (%): 9.3% -> 13.3% -> 15.7% | Improved |
| increase step budget from 20 to 50 steps for model with video pretraining | VideoAgentTrek stage 1 + 2 · OSWorld-Verified | task success rate (%): 14.13 -> 15.78 | Improved |
| increase step budget from 20 to 50 steps for SFT-only model (ablation) | SFT-only (stage 2 only) · OSWorld-Verified | task success rate (%): 9.3 -> 9.27 | No clear effect |
| add SFT on human demonstrations (without video pretraining) to base model | Qwen2.5-VL-7B · AgentNetBench | step success rate: 0.385 -> 0.641 | Improved |
| use ScreenFilter cursor-based filtering of candidate videos | data pipeline · corpus filtering | GUI hours retained: 10,000 hours -> 7,377 hours | Improved |
Closest papers by meaning
- Watch and Learn: Learning to Use Computers from Online Videos (CVPR 2026 · similarity 0.91)
- Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining (ICML 2026 · similarity 0.90)
- Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations (NeurIPS 2025 · similarity 0.87)
- AgentStudio: A Toolkit for Building General Virtual Agents (ICLR 2025 · similarity 0.87)
- VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks (ICLR 2025 · similarity 0.87)
- AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
