Research map / Multi-agent systems and games / LLM agents

Measuring Agents in Production

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Empirical study of production LLM agents combining 20 practitioner interviews and a survey of 306 respondents (86 deployed systems), characterizing models, architectures, evaluation, and challenges.

Problem

Little public information exists on what technical methods make LLM-based agent deployments in production successful, leaving a gap between research advances and real-world practice.

Approach

First systematic, practitioner-sourced characterization of production agents, showing that deployed systems favor simple, controllable designs (prompting frontier models, structured workflows, bounded steps, human evaluation) rather than the autonomy, weight tuning, and automated benchmarks emphasized in research.

Key results

• 68% of deployed agents execute at most 10 steps before human intervention (survey)
• 70% of case studies (14/20) rely on off-the-shelf models without SFT or RL
• 74% of deployed systems rely on human-in-the-loop evaluation; 75% of case studies use no formal benchmark

Surprising findings

• Practitioners avoid RL and post-training despite strong research results, citing brittleness to model upgrades
• Only 3 of 20 case studies use external agent frameworks; 85% build custom in-house implementations

Limitations

• Sampling and participation bias; case teams concentrated in Americas and Europe
• Survey respondents likely skew toward professional networks
• Snapshot of April-November 2025 rather than prevalence estimates

Experimental setup

Datasets: 20 in-depth interview case studies 306-response practitioner survey (86 deployed systems)

Metrics: percentage of respondents 95% bootstrap confidence intervals Cohen's kappa

Builds on: grounded theory DSPy LOTUS

Closest papers by meaning

• TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (NeurIPS 2025 · similarity 0.86)
• General Agent Evaluation (NeurIPS 2026 · similarity 0.86)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox (ICML 2026 · similarity 0.86)
• EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings (ICML 2026 · similarity 0.85)
• Why Do Multi-Agent LLM Systems Fail? (NeurIPS 2025 · similarity 0.85)
• How can we assess human-agent interactions? Case studies in software agent design (ICML 2026 · similarity 0.85)