Research map / Multi-agent systems and games / LLM agents

SCUBA: Salesforce Computer Use Benchmark

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SCUBA (CRMBench), a 300-task benchmark on live Salesforce sandboxes for computer-use agents, with milestone-based evaluation, latency/cost metrics, and demonstration-augmented settings.

Problem

Existing computer-use benchmarks do not capture enterprise CRM workflows on platforms like Salesforce, especially tasks that write or change data and configuration, and do not report latency and cost alongside success.

Approach

A realistic Salesforce sandbox benchmark built from user interviews with rule-based milestone evaluators, task-level config reset, asynchronous parallel evaluation, and human demonstrations distilled from annotated trajectories to improve success while cutting time and cost.

Key results

• Zero-shot, the best browser-use agent GPT-5 reaches 51.33% success, while open-source UI-TARS-1.5-7B reaches 2.67% and OpenCUA-7B 0.67%.
• With human demonstrations, Claude computer-use reaches 47.69% success and Gemini 2.5 Pro (browser-use) reaches 46.15% with lower time, and demonstrations raise success for most agents.
• Computer-use agents have lower latency than browser-use agents but lower success; grounding and planning are main failure causes.

Surprising findings

• Open-source computer-use agents that score well on OSWorld achieve under 5% success on the Salesforce benchmark in zero-shot.
• Demonstrations sometimes increased steps and costs for some agents (e.g., Gemini 2.5 Pro) because agents found shortcuts absent from demonstrations.

Limitations

• Open-source models are tested only at 7B scale and few open computer-use agents are included.
• Browser-use DOM parser needed substantial customization for Salesforce; out-of-box parsing fails on some elements.
• Results depend on a relatively small set of 300 tasks from 60 templates and a single Salesforce developer sandbox.

Experimental setup

Datasets: SCUBA task instances (300 test cases, 60 templates) human annotated trajectories for demonstrations

Benchmarks: SCUBA (CRMBench)

Metrics: milestone score task success rate time (min) steps tokens costs ($)

Baselines: GPT-5 OpenAI Computer-Use Agent Claude computer use UI-TARS-1.5-7B OpenCUA-7B Agent S2.5 Gemini 2.5 Pro Claude Sonnet 4 browser-use

Builds on: OSWorld Browser Use Set-of-Mark prompting vLLM

Closest papers by meaning

• SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows? (NeurIPS 2026 · similarity 0.84)
• ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents (NeurIPS 2026 · similarity 0.83)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation (ICLR 2026 · similarity 0.83)
• EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings (ICML 2026 · similarity 0.83)
• WorkArena: How Capable are Web Agents at Solving Common Knowledge Work Tasks? (ICML 2024 · similarity 0.83)
• DRBench: A Realistic Benchmark for Enterprise Deep Research (ICLR 2026 · similarity 0.82)