Research map / Multi-agent systems and games / LLM agents

SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows?

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agent data synthesis and behavior studies · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SaaS-Bench, a benchmark of 106 long-horizon, cross-application professional tasks across 23 deployable open-source SaaS systems, scored with weighted verification checkpoints; frontier CUAs complete fewer than 4% end-to-end.

Problem

Existing web and GUI agent benchmarks use simplified environments, isolated single-app tasks, and short horizons, so they overestimate computer-use agents' ability to carry out realistic professional SaaS workflows.

Approach

Builds a benchmark on real open-source SaaS deployments (Docker, reset per run) with LLM-assisted task synthesis reviewed by human experts, and grades tasks with weighted state, content, and LLM-judge checkpoints that yield both strict resolved scores and partial checkpoint scores.

Key results

• Best overall checkpoint score is 43.9% (Claude Opus 4.7), with the best resolved score only 3.8%.
• Pass@3 improves over pass@1 by roughly 8 percentage points overall across four representative models.
• Checkpoint scores fall from about 53% on the simplest tasks to under 20% on the longest trajectories.

Surprising findings

• Agents often claim task success while verification shows errors, e.g., a bill date error recognized mid-run but not re-verified and reported as correct.
• Raw trajectory length is a poor proxy for capability: Gemini 3.5 Flash High used the most steps (324) but scored near the bottom.

Limitations

• Only 106 tasks across 23 systems, a relatively small benchmark.
• Only 45% of candidate tasks survived expert review, and tasks depend on LLM-generated candidates.
• Evaluation uses a single browser-use framework, so results may not transfer to other agent scaffolds.

Experimental setup

Datasets: SaaS-Bench tasks (106)

Benchmarks: SaaS-Bench

Metrics: resolved score checkpoint score pass@k steps per passed checkpoint

Baselines: Claude Opus 4.7 GPT-5.5 High Claude Opus 4.6 GPT-5.4 High Kimi K2.6 Qwen 3.6 Plus Gemini 3.1 Pro Doubao Seed 2.0 Pro Claude Sonnet 4.6 DeepSeek V4 Pro GLM-5.1 MiniMax M2.7

Builds on: browser-use WebArena WorkArena

Closest papers by meaning

• WorkArena: How Capable are Web Agents at Solving Common Knowledge Work Tasks? (ICML 2024 · similarity 0.89)
• TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (NeurIPS 2025 · similarity 0.87)
• OpenCUA: Open Foundations for Computer-Use Agents (NeurIPS 2025 · similarity 0.87)
• ProSoftArena: Benchmarking Hierarchical Capabilities of Multi-modal Agents in Professional Software Environments (CVPR 2026 · similarity 0.87)
• Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.87)
• AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use (ICML 2026 · similarity 0.86)