Research map / Multi-agent systems and games / LLM agents
DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DevOps-Gym, a benchmark of ~700 Java/Go tasks across build and configuration, monitoring, issue resolving, test generation, and end-to-end pipelines, and evaluates coding agents and LLMs on it.
Problem
AI agents' capabilities across the full software DevOps cycle (build/configuration, monitoring, issue resolving, test generation) are unmeasured, since existing benchmarks cover mostly development tasks in Python with no tool-calling dynamic environments.
Approach
A tool-augmented, containerized, end-to-end benchmark over real-world Java and Go repositories covering four DevOps stages plus chained pipeline tasks, with semi-automated expert-driven task construction, contamination screening, and stage-specific metrics.
Key results
- Claude Code with Claude-4-Sonnet achieves 51.85% on build and configuration and 20.56% on monitoring, the best results across agents.
- Claude-4-Sonnet under OpenHands reaches 23.87% issue resolving and 11.61% test generation, versus 70.4% on SWE-bench Verified with the same setup.
- All agents achieve 0% full success on the 14 end-to-end pipeline tasks (paper text says 18 in one place).
Surprising findings
- Agents score 0% on end-to-end pipeline tasks, though they sometimes succeed at later stages with incorrect monitoring conclusions.
- Coding agents seek shortcuts during issue reproduction, injecting easier-to-trigger errors instead of reproducing the reported failure.
Limitations
- Monitoring and build tasks are small in number (30 and 48 tasks), and monitoring relies on a few repositories.
- Evaluation covers only a limited set of agents and models, and the paper states it will extend to weaker models.
- Stages such as CI/CD and infrastructure management are excluded due to mutable external systems.
Experimental setup
Datasets: Multi-SWE-bench SWT-bench
Benchmarks: DevOps-Gym
Metrics: success rate fail-to-pass resolve rate monitoring diagnosis accuracy end-to-end pipeline success
Baselines: OpenHands mini-SWE-agent Aider Claude Code
Builds on: SWE-bench Multi-SWE-bench SWT-bench terminal-bench BugSwarm
Closest papers by meaning
- EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings (ICML 2026 · similarity 0.87)
- GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.87)
- Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies (NeurIPS 2026 · similarity 0.85)
- RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.85)
- daVinci-Dev: Agent-native Mid-training for Software Engineering (ICML 2026 · similarity 0.85)
- Benchmarking Agentic Workflow Generation (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.