Research map / Multi-agent systems and games / LLM agents
WebArena: A Realistic Web Environment for Building Autonomous Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds WebArena, a standalone, reproducible, Docker-hosted web environment with four realistic website categories plus tools and knowledge resources, and an 812-task benchmark with functional-correctness evaluation; GPT-4 agents reach 14.41% success versus 78.24% for humans.
Problem
Existing web agent environments over-simplify real-world sites, are static, or evaluate by surface-form action matching rather than functional correctness, so agent results do not transfer to realistic tasks.
Approach
Self-hosted, fully functional replicas of e-commerce, forum, code-hosting, and CMS sites (populated with real-world-derived data) plus utility tools and knowledge bases, with programmatic state-based validators that check whether task outcomes are achieved rather than matching reference action sequences.
Key results
- Best GPT-4 agent (CoT, no UA hint) reaches 14.41% end-to-end success, versus 78.24% human performance.
- GPT-4 with CoT and UA hint achieves 11.70%; GPT-3.5 with CoT reaches 8.75% and text-bison-001 reaches 5.05%.
- GPT-4 wrongly labels 54.9% of feasible tasks as impossible when the UA hint is included, which motivated removing the hint.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add chain-of-thought reasoning | GPT-3.5 with UA hint on WebArena · WebArena | SR: 6.41 -> 8.75; +2.34% | Helped |
| Remove the UA hint (ablation) | GPT-4 with CoT on WebArena · WebArena achievable tasks | SR_AC: 8.63 -> 13.02 | Helped |
| Remove the UA hint (ablation) | GPT-4 with CoT on WebArena · WebArena unachievable tasks | SR_UA: 77.78 -> 44.44 | Hurt |
| Remove the UA hint (ablation) | GPT-4 with CoT on WebArena · WebArena | SR: 11.70 -> 14.41 | Helped |
| Remove the UA hint (ablation) | GPT-3.5 with CoT on WebArena · WebArena | SR: 8.75 -> 6.16 | Hurt |
Surprising findings
- The unachievable-task hint hurts overall performance because models prematurely stop on feasible tasks.
- GPT-4 agents rarely complete all task variants within a template, showing that similar tasks can differ greatly in difficulty.
Limitations
- Agents are evaluated only with few-shot prompting of closed API models; no fine-tuned agents are studied.
- Map data is confined to the northeast US region due to storage constraints.
- Human annotators are computer science graduate students, and human performance may not reflect general users.
Experimental setup
Datasets: WebShop product data dump Reddit subreddit and post data (sampled) GitLab repositories (sampled) English Wikipedia (Kiwix, May 2023) OpenStreetMap
Benchmarks: WebArena benchmark (812 tasks)
Metrics: end-to-end task success rate success rate on achievable tasks success rate on unachievable tasks
Baselines: GPT-4 GPT-3.5 text-bison-001 Human
Builds on: Mind2Web WebShop MiniWoB++ ReAct Chain-of-thought prompting
Closest papers by meaning
- REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites (NeurIPS 2025 · similarity 0.91)
- WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks (NeurIPS 2024 · similarity 0.91)
- WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agents (ICLR 2026 · similarity 0.90)
- Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation (ICLR 2025 · similarity 0.90)
- WorkArena: How Capable are Web Agents at Solving Common Knowledge Work Tasks? (ICML 2024 · similarity 0.89)
- A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.