Research map / Multi-agent systems and games / LLM agents

Computer Use at the Edge of the Statistical Precipice

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that a ~1MB blind replay script can match or beat frontier CUA agents on static benchmarks, proves replay success equals pass@k, and proposes PRISM design principles, the DigiWorld mobile benchmark, and a hierarchical bootstrap for CUA evaluation.

Problem

Existing computer-use agent (CUA) benchmarks use static, unverified environments and flawed evaluation metrics (pass@k, Wald intervals, i.i.d. aggregation), so they can be gamed by memorized action sequences and yield unreliable rankings.

Approach

A replay agent storing one successful trajectory per task achieves expected success equal to the source agent's pass@k on deterministic benchmarks; the paper pairs this diagnosis with PRISM environment design principles (privileged, realistic, integrity-checked, sandboxed, multifactorial) instantiated in a combinatorial mobile benchmark, plus Wilson intervals and a hierarchical bootstrap for aggre

Key results

• A ~1MB replay script achieves 71.1% on OSWorld vs. 70.6% for the frontier source model and 41.5% vs. 32.7% on MobileWorld, but drops to 6.90% on DigiWorld vs. 45.7%.
• Wald intervals cover the true value only 25% of the time at R=3, while Wilson intervals stay near 95%.
• Hierarchical bootstrap reduces expected regret from 9.8% to 6.0% in the Claude vs. Gemini comparison; rollout-only bootstrap reaches 17% suite-level coverage versus 95% for the full hierarchy.

Surprising findings

• A replay agent with no screen observation outperforms its frontier source model on OSWorld and MobileWorld.
• Under heterogeneous conditions, rollout-only bootstrap covers only 17% (or 60% in simulation) of suite-level outcomes.

Limitations

• PRISM does not address harness design, verifier engineering, or sandboxing challenges for complex environments.
• DigiWorld currently targets only Android mobile apps.
• MobileWorld evaluation uses a 96-task subset after excluding 105 tasks.

Experimental setup

Benchmarks: DigiWorld OSWorld MobileWorld

Metrics: success rate pass@k confidence interval coverage expected regret mean absolute deviation

Baselines: OSWorld MobileWorld AndroidWorld Gemini 3 Flash Claude Sonnet 4.6

Builds on: pass@k rliable Wilson score interval

Closest papers by meaning

• OpenApps: Simulating Environment Variations to Measure UI Agent Reliability (ICLR 2026 · similarity 0.88)
• OpenCUA: Open Foundations for Computer-Use Agents (NeurIPS 2025 · similarity 0.88)
• OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (ICLR 2026 · similarity 0.88)
• VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics (ICML 2026 · similarity 0.88)
• AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions (ICML 2026 · similarity 0.87)
• AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use (ICML 2026 · similarity 0.87)