Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Synatra turns indirect knowledge (wikiHow tutorials and ClueWeb web snapshots) into synthetic direct (state, action) web-agent demonstrations via LLM rewriting and observation generation, and fine-tunes CodeLlama-7B on ~100k examples.
Problem
Digital agents lack large-scale direct demonstrations (state-action pairs) for web tasks; human annotation is costly and environment-based collection has limited coverage.
Approach
Re-purpose indirect knowledge (how-to tutorials and random web observations) into program-formatted trajectories by prompting an LLM to write hypothetical tasks and actions, and generating the intermediate HTML/accessibility-tree observations, with filtering via next-state prediction.
Key results
- Synatra-CodeLlama-7b reaches 15.85% step accuracy on Mind2Web, 38.20% on MiniWoB++ and 6.28% on WebArena, outperforming GPT-3.5 on Mind2Web and WebArena and all comparably sized baselines.
- Synthetic demonstrations cost about $0.031 each (~3% of the $0.8 human Mind2Web trajectory cost) and 9k synthetic demos improved WebArena by 4.56% while human demos from Mind2Web gave no WebArena gain.
- Performance on WebArena rises steadily as synthetic data scales from 18k to 100k samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Finetune CodeLlama-instruct-7b on Synatra synthetic demonstrations | CodeLlama-instruct-7b · Mind2Web test | step accuracy (%): 6.62% to 15.85% | Helped |
| Finetune CodeLlama-instruct-7b on Synatra synthetic demonstrations | CodeLlama-instruct-7b · MiniWoB++ | task success rate (%): 23.04% to 38.20% | Helped |
| Finetune CodeLlama-instruct-7b on Synatra synthetic demonstrations | CodeLlama-instruct-7b · WebArena | task success rate (%): 0.00 to 6.28% | Helped |
| Finetune CodeLlama on 9k Synatra demonstrations instead of 9k human demonstrations | CodeLlama · MiniWoB++ | task success rate (%): +17.81% | Helped |
| Finetune CodeLlama on 9k Synatra demonstrations instead of 9k human demonstrations | CodeLlama · WebArena | task success rate (%): +4.56% | Helped |
| Represent trajectories as natural language instead of programs (ablation) | Synatra · MiniWoB++ | task success rate | Hurt |
| Represent trajectories as natural language instead of programs (ablation) | Synatra · WebArena | task success rate | Hurt |
| Use demonstrations from only tutorials or only randomly sampled web pages instead of mixing both sources (ablation) | Synatra · WebArena and MiniWoB++ | task success rate | Hurt |
| Scale synthetic training data from 18k to 100k samples (ablation) | Synatra-CodeLlama-7b · WebArena | task success rate | Helped |
| Add retrieved tutorials as prompt context instead of finetuning on Synatra trajectories (ablation) | Llama3.1-instruct-8B · WebArena | task success rate | Helped |
Surprising findings
- Retrieval-augmented prompting with raw tutorials gives only marginal gains, suggesting models struggle to use indirect knowledge directly.
- Mind2Web human demonstrations improved MiniWoB++ modestly and failed entirely on WebArena, while 9k synthetic demos did better.
Limitations
- Relies on a strong LLM (GPT-4/GPT-3.5) for data generation, which costs money and time.
- Generated observations (HTML) may not reflect real-world web complexity; trajectories may be hallucinated.
- Dependent on quality of source web page data.
Experimental setup
Datasets: wikiHow ClueWeb Mind2Web
Benchmarks: Mind2Web MiniWoB++ WebArena
Metrics: step accuracy task success rate
Baselines: GPT-3.5 GPT-4 CodeLlama-instruct-7b Llama3-chat-8b Llama3-chat-70b AgentLM CodeActAgent AgentFlan Lemur-chat-70b AgentLM-70b FireAct-7b AutoWebGLM-7b
Builds on: WebArena Mind2Web chain-of-thought
Closest papers by meaning
- Learn-by-interact: A Data-Centric Framework For Self-Adaptive Agents in Realistic Environments (ICLR 2025 · similarity 0.86)
- AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning (NeurIPS 2024 · similarity 0.86)
- LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations (ICML 2025 · similarity 0.86)
- Making AI Policies Transparent to Humans through Demonstrations (AAAI 2024 · similarity 0.86)
- RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Users (AAAI 2026 · similarity 0.86)
- VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.