Research map / Multi-agent systems and games / LLM agents
AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Synthesizes GUI/web agent trajectories by harvesting web tutorials from RedPajama, filtering them with a pre-filter, GPT-4o-mini labeler and FastText classifier, then replaying them with a GPT-4o VLM agent in BrowserGym and filtering with a GPT-4o evaluator, yielding 10,398 trajectories at about $0.55 each.
Problem
GUI and web agents lack high-quality multi-step trajectory data, and human annotation is too costly to scale.
Approach
Use publicly available web tutorials as guidance for a VLM agent that replays them in live browser environments, with a VLM evaluator filtering successful trajectories, producing multimodal agent data at low cost without human annotators.
Key results
- Qwen2.5-32B-Instruct with AgentTrek reaches 22.40% on WebArena, versus 13.10% for GPT-4o and 14.41% for GPT-4.
- Qwen2-VL-7B with AgentTrek reaches 67.4 average on ScreenSpot Web grounding, above GPT-4 + OmniParser at 67.0.
- The pipeline costs about $0.551 per verified trajectory, with replay at $215.36 per 1k tasks using GPT-4o.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2.5-7B-Instruct on AgentTrek textual trajectories | Qwen2.5-7B-Instruct · WebArena | task success rate: 3.80 -> 10.46 | Helped |
| Fine-tune Qwen2.5-32B-Instruct on AgentTrek textual trajectories | Qwen2.5-32B-Instruct · WebArena | task success rate | Helped |
| Fine-tune Qwen2-VL-7B on AgentTrek trajectories | Qwen2-VL-7B · ScreenSpot Web | Average grounding performance: 30.7 -> 67.4 | Helped |
| Train Qwen2-VL with AgentTrek data | Qwen2-VL · Multimodal-Mind2Web Cross-Domain | Step SR | Helped |
| Combine AgentTrek and Mind2Web training data | Qwen2-VL · Multimodal-Mind2Web | Step SR | Helped |
| Increase AgentTrek training data from 20% to 100% (ablation) | Qwen2-VL · Multimodal-Mind2Web Cross-Domain | Step SR: 39.5% to 45.0% | Helped |
| Follow detailed tutorial instructions instead of only a high-level goal | Guided replay agent · Guided replay tasks | success rate: 15.78% -> 52% | Helped |
Surprising findings
- Tutorial-guided replay raised agent success from 15.78% to 52% on 400 tasks compared with goal-only guidance.
- The LLM labeler reportedly caught tutorial content in longer texts that human annotators missed.
Limitations
- Tutorials can expire, causing replay failures such as booking before the actual date.
- Coverage limited to 11 task categories and 127 websites.
- Evaluator accuracy is only 84.0% on the human-annotated validation set.
Experimental setup
Datasets: RedPajama AgentTrek synthesized trajectories Mind2Web
Benchmarks: WebArena ScreenSpot Multimodal-Mind2Web
Metrics: task success rate grounding accuracy Ele.Acc Op.F1 Step SR
Baselines: GPT-4o GPT-4 GPT-4V LLaMa3-chat-70B LLaMa3-chat-8B Synatra-CodeLlama-7B AutoWebGLM SeeClick CogAgent GPT-4 + OmniParser Mind2Web-trained Qwen2-VL
Builds on: BrowserGym RedPajama Qwen2-VL Qwen2.5 FastText
Closest papers by meaning
- HATS: Hardness-Aware Trajectory Synthesis for GUI Agents (CVPR 2026 · similarity 0.88)
- WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agents (ICLR 2026 · similarity 0.88)
- AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents (ICLR 2026 · similarity 0.88)
- AgentStudio: A Toolkit for Building General Virtual Agents (ICLR 2025 · similarity 0.88)
- TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents (AAAI 2026 · similarity 0.88)
- Agent S: An Open Agentic Framework that Uses Computers Like a Human (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.