Research map / Multi-agent systems and games / LLM agents
How to Train Your LLM Web Agent: A Statistical Diagnosis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Statistical study of 1,370 training configurations for LLM web agents, showing that branching from SFT checkpoints into GRPO RL early (but not immediately) gives better compute-performance trade-offs on MiniWoB++ and WorkArena.
Problem
Open-source LLM web agents lag proprietary ones, and RL/SFT training pipelines are costly and sensitive to hyperparameters and seeds, so the compute allocation between expert demonstrations and on-policy RL is unclear.
Approach
Bootstrap-based hyperparameter importance analysis over a large random sweep of SFT+GRPO runs, with RL branched from multiple SFT checkpoints, to derive compute-aware recipes for training web agents.
Key results
- SFT+RL on Llama-3.1-8B reaches 67.2% held-out goal success on MiniWoB++ vs 55.2% for SFT alone and 63.2% for the Llama-3.3-70B teacher.
- Early RL branching matches peak pure-SFT performance on MiniWoB++ with about 6 vs 11 exaFLOPs (roughly 45% compute saved).
- On WorkArena, Llama-3.1-8B SFT+RL reaches 35.4% held-out goal success vs 28.4% for SFT alone, still below the 36.0% teacher and 52.5% Claude-3.5-Sonnet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Branch into RL early, but not immediately, after SFT instead of continuing pure SFT | Llama-3.1-8B · MiniWoB++ held-out goals and held-out tasks | success rate: 45% less compute; 11 vs 6 exaFLOPs | Helped |
| Add an RL phase after SFT rather than using pure SFT | Llama-3.1-8B · WorkArena | success rate | Helped |
| Add curriculum learning (ablation) | RL from the base Instruct model · Held-out tasks | success rate | Helped |
| Add curriculum learning (ablation) | RL from an SFT-warm-started model · Held-out tasks | success rate | Hurt |
| Add error log feedback (ablation) | RL from the base Instruct model · Held-out tasks | success rate | Helped |
| Add error log feedback (ablation) | RL from an SFT-warm-started model · Held-out tasks | success rate | No clear effect |
| Set decoding temperature to 0.25 (ablation) | RL training configurations · Held-out tasks | success rate | Helped |
| Use grouped-relative advantage instead of raw rewards (ablation) | RL from an SFT-warm-started model · Held-out tasks | success rate | Helped |
| Use grouped-relative advantage instead of raw rewards (ablation) | RL from the base Instruct model · Held-out tasks | success rate | Hurt |
| Enable zero-advantage filtering (ablation) | GRPO training · Held-out tasks | success rate | Helped |
| Normalize advantages by standard deviation (ablation) | RL from less-finetuned models · Held-out tasks | success rate | Helped |
| Normalize advantages by standard deviation (ablation) | RL from heavily warm-started models · Held-out tasks | success rate | Hurt |
Surprising findings
- Curriculum learning helps when RL starts from scratch but becomes detrimental after SFT warm-start.
- Decoding temperature and learning rate choices shift optimal settings with SFT compute; zero-advantage filtering consistently helps while standard-deviation normalization benefit decreases with more SFT.
Limitations
- Results limited to English-language web interfaces and Llama 3 models in the 8B-70B range.
- Random search does not cover the full hyperparameter space; conclusions may change with more exhaustive search.
- WorkArena performance plateaus around 40% with 14 of 33 tasks never solved.
Experimental setup
Datasets: MiniWoB++ WorkArena
Benchmarks: MiniWoB++ WorkArena
Metrics: average task success rate compute in exaFLOPs win rate in bootstrap hyperparameter analysis
Baselines: Claude-3.5-Sonnet GPT-4o GPT-4o-Mini Llama-3.1-70B-Instruct o1-Mini Llama-3.1-405B-Instruct Llama-3.3-70B Instruct Qwen-2.5-72B Instruct
Builds on: GRPO PPO WebRL BrowserGym AgentLab
Closest papers by meaning
- WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning (ICLR 2025 · similarity 0.92)
- Offline Training of Language Model Agents with Functions as Learnable Weights (ICML 2024 · similarity 0.91)
- WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agents (ICLR 2026 · similarity 0.91)
- A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (ICLR 2024 · similarity 0.90)
- WebGym: Scaling Training Environments for Long-Horizon Visual Web Agents with Realistic Tasks (CVPR 2026 · similarity 0.90)
- From History to State: Constant-Context Skill Learning for LLM Agents (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.