Research map / Multi-agent systems and games / LLM agents
ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ITBench, a benchmark framework with 94 real-world IT automation scenarios (SRE, CISO, FinOps) in executable environments, and reports low success rates for baseline LLM agents.
Problem
Measuring how effectively AI agents can automate real-world IT operations tasks (incident resolution, compliance assessment, cost management) before production deployment lacks systematic, realistic, and automatically evaluated benchmarks.
Approach
A extensible benchmark framework that instantiates real-world IT problems as executable, fault-injected environments (scenario = metadata, environment, triggering events, desired outcome) with partial-credit metrics and a leaderboard, evaluated with baseline ReAct/CrewAI-based agents across multiple LLMs.
Key results
- GPT-4o-based SRE agent achieves 13.81% pass@1 on diagnosis and 11.43% on mitigation across 42 SRE scenarios.
- Best CISO compliance agent (GPT-4o) reaches 25.19%-40.28% scenario pass@1 depending on class; GPT-4o-mini achieves 25.19% overall O/A pass@1.
- Agents resolve 0% of FinOps scenarios for mitigation; GPT-4o diagnoses 33% of FinOps scenarios.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove trace data (ablation) | LUMYN with GPT-4o · SRE diagnosis | pass@1 (%): 13.81% -> 9.52% | Hurt |
| remove trace data (ablation) | LUMYN with GPT-4o · SRE mitigation | pass@1 (%): 11.43% -> 2.86% | Hurt |
| increase scenario complexity from Easy to Medium to Hard | LUMYN with GPT-4o · SRE scenarios | mitigation pass@1 (%): 21% -> 12.27% -> 0.0% | Hurt |
| use GPT-4o instead of other evaluated models | LUMYN · SRE diagnosis | pass@1 (%): 13.81 ± 1.67 | Helped |
| use GPT-4o instead of other evaluated models | FinOpsAgent · FinOps diagnosis | pass@1 (%): 33 | Helped |
Surprising findings
- Granite-3.1-8B without specialized finetuning outperforms Llama-3.3-70B on the SRE diagnosis task.
- Agent outcomes are non-deterministic across runs even with hyperparameter tuning, with GPT-4o diagnosing scenario 8 in only 1 of 10 runs.
Limitations
- Only 94 scenarios, with just 2 FinOps scenarios.
- Agent results show substantial run-to-run variance from non-deterministic environments and LLMs.
- Only event-triggered reactive scenarios are covered.
Experimental setup
Datasets: real-world SRE incidents from IBM SaaS products CIS benchmarks FinOps Foundation KPIs
Benchmarks: ITBench SRE scenarios ITBench CISO scenarios ITBench FinOps scenarios
Metrics: pass@1 NTAM mean time to diagnosis mean time to repair time to process proximity to optimal cost/efficiency
Baselines: AIOpsLab TrainTicket InsightBench TSB-AD
Builds on: ReAct CrewAI AIOpsLab SWE-bench
Closest papers by meaning
- CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (ICLR 2026 · similarity 0.85)
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.84)
- PaperBench: Evaluating AI’s Ability to Replicate AI Research (ICML 2025 · similarity 0.84)
- BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows (NeurIPS 2026 · similarity 0.84)
- TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (NeurIPS 2025 · similarity 0.84)
- BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.