Research map / Multi-agent systems and games / LLM agents
ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ST-WebAgentBench, a 375-task BrowserGym-based benchmark with 3,057 safety/trust policies across six dimensions, and the CuP metric, showing three web agents keep only about two-thirds of their completions once policies are enforced.
Problem
Web agent benchmarks score only task completion and ignore whether agents act safely and comply with organizational, user, and task policies in enterprise settings.
Approach
Attach concrete ST policy templates (consent, boundary, strict execution, hierarchy, robustness, error handling) to web tasks and score agents with Completion-under-Policy (CuP), which credits a completion only if no policy is violated, plus a Risk Ratio per dimension and a vision-vs-DOM modality challenge.
Key results
- Across three agents, average CR is 24.3% while CuP falls to 15.0%, roughly a 38% relative drop.
- AWM reaches 46.9% PCR but only 20% CuP with 37 consent breaches; WorkArena-Legacy gets 26% CR and 15% CuP; WebVoyager gets 12.8% CR and 10.3% CuP.
- CuP declines from 18.2% with one active policy to 7.1% with more than five (Spearman rho = -0.71, p<0.001).
Surprising findings
- Higher task completion does not imply safer behavior: AWM has the highest PCR but among the lowest policy compliance.
- Strict all-pass@3 is very low (AWM 5.0%, WebVoyager 2.0%, WorkArena 3.0%), indicating run-to-run brittleness.
Limitations
- Only three open agents evaluated, all sharing a GPT-4o backbone; no proprietary computer-use stacks.
- Tasks cover three applications in English only and are an early slice of enterprise workflows.
- Robustness checks focus on prompt injection; user consent is simulated with auto-approval.
Experimental setup
Datasets: WebArena GitLab tasks WebArena ShoppingAdmin tasks SuiteCRM tasks
Benchmarks: ST-WebAgentBench
Metrics: completion rate CuP pCuP partial completion rate risk ratio all-pass@k
Baselines: AgentWorkflowMemory WorkArena-Legacy WebVoyager
Builds on: WebArena BrowserGym WorkArena AutoGen LangGraph
Closest papers by meaning
- OpenAgentSafety: A Comprehensive Framework For Evaluating Real-World AI Agent Safety (ICLR 2026 · similarity 0.90)
- REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites (NeurIPS 2025 · similarity 0.89)
- SafeArena: Evaluating the Safety of Autonomous Web Agents (ICML 2025 · similarity 0.89)
- Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents (NeurIPS 2026 · similarity 0.87)
- TimeWarp: Evaluating Web Agents by Revisiting the Past (NeurIPS 2026 · similarity 0.87)
- It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.