Research map / Multi-agent systems and games / LLM agents
EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EnterpriseOps-Gym, a containerized enterprise benchmark (1,150 expert-curated tasks, 8 domains, 164 DB tables, 512 tools) evaluating 14 LLMs on stateful, policy-constrained agentic planning and tool use.
Problem
Existing agent benchmarks treat tool calls as stateless or confine enterprise evaluation to single vendors with shallow databases, short horizons and few policy constraints, leaving long-horizon, policy-governed enterprise workflows under-evaluated.
Approach
A high-fidelity, containerized enterprise sandbox with relational databases and hundreds of tools, expert-authored tasks verified by SQL outcome checks (goal, integrity, permission, side effects), infeasible-task refusal tests, and plan-conditioned ablations that separate planning from execution.
Key results
- Claude Opus 4.5 achieves the best average task completion at 37.4% under oracle tool access.
- Human-authored plans improve model success by 14-35 percentage points across models and domains.
- The best model (GPT-5.2 Low) refuses infeasible tasks without side effects only 53.9% of the time.
Surprising findings
- Adding up to 15 distractor tools slightly increased average completion rather than hurting it.
- Task decomposition into subtasks regressed performance in CSM and HR relative to a single ReAct agent.
Limitations
- Evaluation uses oracle tool retrieval, so real retrieval difficulty is not measured.
- Infeasible-task set is small (30 tasks).
Experimental setup
Datasets: EnterpriseOps-Gym tasks
Benchmarks: EnterpriseOps-Gym
Metrics: pass@1 task completion infeasible refusal rate verifier pass rate
Baselines: Claude Opus 4.5 Claude Sonnet 4.5 Gemini-3-Pro Gemini-3-Flash GPT-5 GPT-5.2 DeepSeek-V3.2 GPT-OSS-120B Kimi-K2-Thinking Qwen3-235B
Builds on: ReAct tau-bench WorkArena CRMArena
Closest papers by meaning
- DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle (ICLR 2026 · similarity 0.87)
- Synthesis Through Simulation: Generating Coherent Enterprise Data via Scalable Agent-System Interaction (NeurIPS 2026 · similarity 0.87)
- WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks (NeurIPS 2024 · similarity 0.87)
- Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents (NeurIPS 2026 · similarity 0.87)
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.86)
- Benchmarking Agentic Workflow Generation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.