Research map / Multi-agent systems and games / LLM agents
WorkArena: How Capable are Web Agents at Solving Common Knowledge Work Tasks?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WorkArena, a 33-task enterprise benchmark on the ServiceNow platform, and BrowserGym, a browser-agent environment, and evaluates GPT-4o, GPT-3.5 and Llama3-70B zero-shot web agents on it.
Problem
Existing web-agent benchmarks focus on toy or consumer websites and do not measure how well LLM agents perform common knowledge-work tasks in enterprise software.
Approach
A remote-hosted enterprise benchmark of 33 ServiceNow tasks with natural-language goals, programmatic validation and oracle solutions, paired with a unified browser environment (BrowserGym) offering multimodal observations, code/high-level actions and chat-based interaction.
Key results
- GPT-4o agent reaches 42.7% success rate on WorkArena versus 6.1% for GPT-3.5 and 17.9% for Llama3-70B.
- GPT-4o achieves 23.5% on WebArena, reported as the best zero-shot result at the time versus 14.4% for GPT-4 in the original paper.
- List-filter tasks score 0% for all LLMs despite being simple for humans.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add chain-of-thought (use_thinking) (ablation) | GPT-3.5 and Llama3 agents · WorkArena | success rate: GPT-3.5 8.5 to 6.1 when removed; Llama3 20.0 to 8.5 when removed | Helped |
| add multi_actions (multiple actions per step) (ablation) | GPT-4o · WorkArena | success rate: 45.5 to 40.6 | Hurt |
| add coordinate features and bid+coord action set (ablation) | GPT-4o · WorkArena | success rate: 45.5 to 41.2 (MiniWoB 68.2 to 72.6) | Hurt |
| add use_think_history (memory of past thoughts) (ablation) | GPT-4o and GPT-3.5 · WorkArena | success rate: GPT-4o 45.5 to 42.4; GPT-3.5 8.5 to 3.0 | Hurt |
| add use_error_history (ablation) | GPT-4o · WorkArena | success rate: 45.5 to 43.6 | No clear effect |
| add multi_actions (ablation) | GPT-3.5 · WorkArena | success rate: 8.5 to 9.4 | No clear effect |
| add use_think_history memory to Llama3 (removal ablation) (ablation) | Llama3-70b · WorkArena | success rate: 20.0 to 18.8 when removed (i.e., removing it slightly lowers score) | Hurt |
| add multi_actions (ablation) | Llama3-70b · WorkArena | success rate: 20.0 to 17.6 | No clear effect |
| use screenshot with Set-of-Mark (GPT-4o-V) versus text-only GPT-4o | GPT-4o · WorkArena | success rate: 42.7 to 41.8 | No clear effect |
| use open-source Llama3-70b instead of GPT-3.5 / GPT-4o | WorkArena · WorkArena | success rate: Llama3 17.9 vs GPT-3.5 6.1 but GPT-4o 42.7 | Mixed |
| add use_action_history (ablation) | GPT-3.5 and Llama3 agents · WorkArena | success rate: GPT-3.5 8.5 to 5.2 when removed; Llama3 20.0 to 8.5 when removed | Helped |
Surprising findings
- Adding GPT-4o vision with Set-of-Mark gives only minor gains, suggesting current VLMs lack screen-related training.
- Extra features such as long action descriptions or memory (think history) often degrade performance for weaker LLMs and GPT agents.
Limitations
- Experiments limited to three LLMs and a single zero-shot agent design.
- Agents use only AXTrees on WorkArena because HTML is prohibitively large.
- Benchmark tied to the ServiceNow platform and BrowserGym does not provide memory.
Experimental setup
Datasets: WorkArena MiniWoB WebArena
Benchmarks: WorkArena MiniWoB WebArena
Metrics: success rate standard error
Baselines: GPT-3.5 Llama3-70B GPT-4o-V WebArena GPT-4 agent
Builds on: WebArena MiniWoB WebShop Set-of-Mark prompting Playwright
Closest papers by meaning
- WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks (NeurIPS 2024 · similarity 0.93)
- WebArena: A Realistic Web Environment for Building Autonomous Agents (ICLR 2024 · similarity 0.89)
- ProSoftArena: Benchmarking Hierarchical Capabilities of Multi-modal Agents in Professional Software Environments (CVPR 2026 · similarity 0.89)
- TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (NeurIPS 2025 · similarity 0.89)
- SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows? (NeurIPS 2026 · similarity 0.89)
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.