Research map / Multi-agent systems and games / LLM agents
WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WebRL, a self-evolving online curriculum RL framework that trains open Llama-3.1 and GLM-4 models into web agents on WebArena-Lite using an LLM-based outcome reward model and a KL-constrained off-policy update with a replay buffer.
Problem
Open LLMs lack decision-making ability for web agents, and online RL on WebArena suffers from scarce training tasks, sparse outcome-only rewards, and policy drift/forgetting during online learning.
Approach
Generate new curriculum tasks by evolving previously failed instructions with GPT-4o and filtering by critic score, train an LLM outcome-supervised reward model to label rollouts, and update the policy with a KL-constrained squared-advantage loss plus a perplexity-filtered replay buffer of successful trajectories.
Key results
- Llama3.1-8B success rate rises from 4.8% to 42.4% on WebArena-Lite with WebRL, above GPT-4-Turbo (17.6%) and AutoWebGLM (18.2%).
- GLM-4-9B with WebRL reaches 43.0% average success rate.
- Llama3.1-70B with WebRL reaches 49.1%, a 26.1% improvement over its SFT baseline (23.0%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train GLM-4-9B with WebRL instead of using GLM-4-Chat | GLM-4-9B · WebArena-Lite | Avg. SR: 6.1 -> 43.0 | Helped |
| Train Llama3.1-8B with WebRL instead of using Llama3.1-Instruct | Llama3.1-8B · WebArena-Lite | Avg. SR: 4.8 -> 42.4 | Helped |
| Train Llama3.1-70B with WebRL instead of SFT (BC) | Llama3.1-70B · WebArena-Lite | Avg. SR: 23.0 -> 49.1 | Helped |
| Use WebRL instead of DigiRL | GLM-4-9B · WebArena-Lite | Avg. SR: 31.5 -> 43.0 | Helped |
| Use WebRL instead of DigiRL | Llama3.1-8B · WebArena-Lite | Avg. SR: 30.3 -> 42.4 | Helped |
| Use WebRL instead of GPT-4-Turbo | Llama3.1-70B · WebArena-Lite | Avg. SR: 17.6 -> 49.1 | Helped |
| Remove the replay buffer (ablation) | WebRL · WebArena-Lite | performance over training | Hurt |
| Remove the KL-constrained policy update and use REINFORCE with a value baseline (ablation) | WebRL · WebArena-Lite | performance | Hurt |
| Remove both KL-constrained updates and the replay buffer (ablation) | WebRL · WebArena-Lite | performance over training | Hurt |
| Remove self-evolving curriculum learning and keep only first-phase instructions (ablation) | WebRL · WebArena-Lite | performance | Hurt |
| Use only very low-perplexity replay data [1, 1/0.95] instead of moderately difficult data (ablation) | WebRL · WebArena-Lite, first learning phase | performance: 31.5 -> 27.9 | Hurt |
| Use only high-perplexity replay data [1/0.5, ∞] instead of moderately difficult data (ablation) | WebRL · WebArena-Lite, first learning phase | performance: 31.5 -> 23.0 | Hurt |
Surprising findings
- Filtering replay data to moderate perplexity works best; training only on very familiar or very unfamiliar data hurts performance.
- Standard REINFORCE with value baseline overfits the current phase and falls below its initial performance without KL control.
Limitations
- Relies on GPT-4o for curriculum instruction generation and infeasible-task filtering, so not fully proprietary-free.
- Performance on the Map site declines in later phases, indicating trade-offs across websites.
- Evaluated on the 165-case WebArena-Lite subset rather than full WebArena for the main comparison.
Experimental setup
Datasets: WebArena-Lite training set GPT-4o-generated curriculum instructions rollouts from baselines for ORM training
Benchmarks: WebArena-Lite
Metrics: task success rate
Baselines: GPT-4-Turbo GPT-4o AWM WebPilot AutoWebGLM GLM-4-Chat Llama3.1-Instruct SFT (BC) Filtered BC AWR DigiRL
Builds on: DigiRL AWR GAE DPO in-breadth evolving (WizardLM)
Closest papers by meaning
- How to Train Your LLM Web Agent: A Statistical Diagnosis (NeurIPS 2025 · similarity 0.92)
- WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agents (ICLR 2026 · similarity 0.91)
- WebGym: Scaling Training Environments for Long-Horizon Visual Web Agents with Realistic Tasks (CVPR 2026 · similarity 0.90)
- A Subgoal-driven RL Framework for Improving Long-Horizon Web Agents (NeurIPS 2026 · similarity 0.90)
- SeRL: Self-play Reinforcement Learning for Large Language Models with Limited Data (NeurIPS 2025 · similarity 0.89)
- Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.