Research map / Multi-agent systems and games / LLM agents
REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces REAL, a benchmark of 11 deterministic, hosted simulations of popular websites with 112 tasks, scored by localStorage state checks and LLM-judged rubrics, and evaluates frontier models as web agents.
Problem
Web agent evaluation on live websites is non-reproducible, cannot easily test edge cases, risks real state changes, and is hard to use for RL training; prior simulated benchmarks are artificial or require local hosting.
Approach
Deterministic, publicly hosted high-fidelity replicas of 11 popular consumer websites built with React/Next.js, with URL-parameter configurability (latency, error modes, accessibility), localStorage-based state-diff verification for action tasks, and LLM-judged rubrics for retrieval tasks, with a flexible harness supporting Playwright, CDP, and black-box agents.
Key results
- Claude 3.7 Sonnet Thinking achieves the top success rate of 41.07% on the 112 REAL tasks, followed by Gemini 2.5 Pro Experimental at 38.39% and o3 at 34.82%.
- GPT-4o reaches 14.29%, and open models lag (Llama-4-Maverick 12.50%, Llama 3.3 70B 10.71%, DeepSeek V3 19.64%).
- OpenAI's Computer-Using Agent scores only 7.14%, frequently distracted and failing to complete final steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Claude 3.7 Sonnet-Thinking instead of the other evaluated models | REAL baseline agent · REAL (112 tasks) | task success rate: 41.07\% vs 38.39\% | Helped |
| Use reasoning models rather than standard pre-trained models | REAL baseline agent · REAL (112 tasks) | task success rate: o3 (34.82\%), o3-mini (25.00\%), and o1 (16.07\%) | Helped |
| Scale from Llama 3.3 70B to Llama 4 Maverick | REAL baseline agent · REAL (112 tasks) | task success rate: 10.71\% -> 12.50\% | No clear effect |
| Use DeepSeek V3 instead of the evaluated Llama models | REAL baseline agent · REAL (112 tasks) | task success rate: 19.64\% vs 12.50\% and 10.71\% | Helped |
| Use a smaller model | REAL baseline agent · REAL (112 tasks) | task success rate: 1.79\%, 2.68\%, and 9.82\% | Hurt |
| Use OpenAI CUA | REAL baseline agent · REAL (112 tasks) | task success rate: 7.14\% | Hurt |
Surprising findings
- OpenAI's Computer-Using Agent scored only 7.14%, below several text-based models evaluated through the default agent.
- Llama-4-Maverick performed roughly on par with the much older Llama 3.3 70B, suggesting scale alone did not improve web navigation.
Limitations
- Only 11 environments and 112 tasks, a finite and small suite.
- Focused solely on web-based interactions rather than broader agent applications.
- Binary outcome rewards only; dense step-wise rewards not yet provided.
Experimental setup
Benchmarks: REAL
Metrics: end-to-end task success rate
Baselines: WebArena VisualWebArena WorkArena AgentQ AgentOccam WebPilot WebDreamer OpenAI Computer-Using Agent
Builds on: WebArena BrowserGym MiniWoB++ WebShop Mind2Web
Closest papers by meaning
- AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web (ICML 2026 · similarity 0.92)
- WebArena: A Realistic Web Environment for Building Autonomous Agents (ICLR 2024 · similarity 0.91)
- ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents (NeurIPS 2026 · similarity 0.91)
- Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge (NeurIPS 2025 · similarity 0.90)
- TimeWarp: Evaluating Web Agents by Revisiting the Past (NeurIPS 2026 · similarity 0.90)
- AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.