Research map / Multi-agent systems and games / LLM agents

Towards a Science of AI Agent Reliability

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a twelve-metric reliability framework (consistency, robustness, predictability, safety) for AI agents and evaluates 15 frontier models on GAIA and a verified τ-bench subset, finding that reliability has improved only slightly despite capability gains.

Problem

Agent evaluation relies on mean task success, which hides run-to-run variance, sensitivity to perturbations, poor confidence calibration, and severity of failures, so accuracy gains do not translate into reliable deployment.

Approach

Adapts safety-critical engineering reliability principles into four dimensions and operationalizes them as twelve metrics that are independent of raw accuracy, including multi-run consistency, fault/environment/prompt perturbation robustness, calibration/discrimination/Brier predictability, and LLM-judged compliance and harm severity.

Key results

• Overall reliability shows only small improvements over 24 months of frontier model releases, with GAIA showing barely any improvement.
• Outcome consistency remains low across all models, and prompt robustness is a key differentiator across models.
• Calibration has improved noticeably in recent models while discrimination on GAIA has not improved and even worsened in some recent models.

Surprising findings

• Smaller models frequently achieve equal or higher consistency than larger models, suggesting larger models' multiple solution paths increase run-to-run variability.
• Models handle genuine technical faults gracefully yet remain vulnerable to surface-level prompt paraphrasing.

Limitations

• Only two benchmarks and a single scaffold per benchmark.
• Safety evaluation relies on LLM-based judging with its own reliability concerns.
• Metric and aggregation choices are subjective; temperature set to zero may overestimate achievable reliability.

Experimental setup

Datasets: GAIA validation (165 tasks) tau-bench verified airline subset (26 tasks)

Benchmarks: GAIA tau-bench

Metrics: outcome consistency trajectory consistency resource consistency fault robustness environment robustness prompt robustness calibration (ECE) AUROC discrimination Brier score compliance

Baselines: GPT-4 Turbo GPT-4o mini o1 GPT-5.2 Gemini 2.5 Pro Claude Opus 4.5 Claude Sonnet 4

Builds on: ReAct tau-bench GAIA pass^k

Closest papers by meaning

• Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability (NeurIPS 2026 · similarity 0.95)
• Agentic Confidence Calibration (ICML 2026 · similarity 0.90)
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (ICLR 2026 · similarity 0.89)
• OpenAgentSafety: A Comprehensive Framework For Evaluating Real-World AI Agent Safety (ICLR 2026 · similarity 0.89)
• Automata from Agent Traces: Failure and Next-Step Prediction (NeurIPS 2026 · similarity 0.89)
• Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents (NeurIPS 2026 · similarity 0.89)