Research map / Large language models / Agents, code and math
WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WorkArena++, a 682-task benchmark of compositional enterprise knowledge-work workflows on ServiceNow, evaluating LLM and VLM web agents against humans, with agents largely failing while humans score high.
Problem
Web agents' planning, reasoning, retrieval and contextual understanding on realistic, multi-step enterprise knowledge-work tasks remain underexplored; existing benchmarks are atomic or synthetic.
Approach
Compose verified oracle/validator building blocks from WorkArena L1 into compositional tasks across five skill categories at two difficulty levels (explicit L2 instructions and implicit ticket-based L3), with seeded reproducible curricula and automatic generation of oracle observation-action traces for fine-tuning.
Key results
- GPT-4o achieves 0% on L3 tasks and about 3.0% on L2 tasks in the agent curriculum, while humans reach 93.9% on the human curriculum subset.
- GPT-4o on the human-evaluated subset scores 2.1% overall versus 93.9% for 15 human participants.
- GPT-4o-v (vision) reaches 3.8% on L2 vs 3.0% for text-only GPT-4o, and only GPT-4o variants show nonzero success on L2 memorization tasks (14.6%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o instead of GPT-3.5 | WorkArena++ agent · WorkArena++ L2 | success rate: 0.0 -> 3.0 | Helped |
| Provide screenshots with vision | GPT-4o agent · WorkArena++ L2 information retrieval | success rate: 0.0 -> 3.6 | Helped |
| Provide screenshots with vision | GPT-4o agent · WorkArena++ overall L2 | success rate: 3.0 -> 3.8 | Helped |
| Provide screenshots with vision | GPT-4o agent · WorkArena L1 | success rate: 42.7 -> 41.8 | Hurt |
| Provide screenshots with vision | GPT-4o agent · MiniWoB | success rate: 71.3 -> 72.5 | Helped |
| Provide screenshots with vision | GPT-4o agent · WebArena | success rate: 23.5 -> 24.0 | Helped |
| Use the L3 ticket plus knowledge-base instruction format instead of explicit L2 instructions | GPT-4o agent · WorkArena++ | success rate: 3.0 -> 0.0 | Hurt |
| Use humans instead of GPT-4o | WorkArena++ human curriculum · WorkArena++ L2 | success rate: 2.1 -> 93.9 | Helped |
Surprising findings
- Agents that score reasonably on WebArena and MiniWoB nearly fail entirely on WorkArena++, with zero success on all L3 tasks across models.
- Human participants with brief training (15 min video plus 15 min exploration) solved the tasks at 93.9%, far above all LLM agents.
Limitations
- Benchmark covers only the ServiceNow platform and does not include tasks spanning other software.
- Human cohort is not representative of general ServiceNow users, with many ServiceNow employees and advanced-degree holders.
- Only a limited set of LLMs and VLMs was evaluated; models with very long context such as Gemini 1.5 Pro were not included.
Experimental setup
Datasets: WorkArena L1 tasks (33)
Benchmarks: WorkArena++ WorkArena L1 WebArena MiniWoB
Metrics: success rate
Baselines: GPT-3.5 GPT-4o GPT-4o-v Llama3-70b Mixtral-8x22b human evaluators
Builds on: WorkArena BrowserGym WebArena chain-of-thought WebVoyager
Closest papers by meaning
- WorkArena: How Capable are Web Agents at Solving Common Knowledge Work Tasks? (ICML 2024 · similarity 0.93)
- WebArena: A Realistic Web Environment for Building Autonomous Agents (ICLR 2024 · similarity 0.91)
- TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (NeurIPS 2025 · similarity 0.87)
- ProSoftArena: Benchmarking Hierarchical Capabilities of Multi-modal Agents in Professional Software Environments (CVPR 2026 · similarity 0.87)
- LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agents (ICLR 2024 · similarity 0.87)
- AOrchestra: Automating Sub-Agent Creation for Agentic Orchestration (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.