Research map / Multi-agent systems and games / LLM agents
$\chi$-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces χ-Bench, a high-fidelity simulated healthcare operations benchmark (PA, UM, CM) with 20 apps via 87 MCP tools and a 1,279-document handbook skill, showing frontier agents resolve at most 28% of tasks.
Problem
Current benchmarks do not test whether AI agents can automate end-to-end, long-horizon, policy-dense healthcare administrative workflows involving multi-role handoffs and multi-turn dialogues with humans.
Approach
A simulator-based benchmark where agents drive full prior-authorization, utilization-management, and care-management cases through MCP tools, guided by a large managed-care operations handbook skill, scored in-situ by a deterministic contract plus rubric LLM judge.
Key results
- Best configuration Claude Code + Claude Opus 4.6 resolves 28.0% of tasks at pass@1; no configuration exceeds 20% at pass^3.
- Single-session marathon run drops performance to 3.8% overall.
- End-to-end two-agent PA arena drops Codex + GPT-5.5 from 30.4% to 0% pass@1.
Surprising findings
- Trimming the handbook skill improves PA performance in the -Both setting, as agents with a handbook over-verify and refuse to submit.
- Re-surfacing MCP tools as CLI commands is neutral-to-worse rather than beneficial.
Limitations
- Language-only agents; no multimodal imaging or speech tasks.
- Covers only three workflow domains; hundreds of long-tail healthcare workflows remain uncovered.
- Opus 4.7 is the only judge model; effects of other judges not studied.
Experimental setup
Datasets: Managed-Care Operations Handbook (1,279 documents) simulated patient charts (~5,000 chart activities, 50 patients)
Benchmarks: χ-Bench
Metrics: pass@1 pass@3 pass^3 cost steps marathon pass@1 E2E arena pass@1
Baselines: HealthAdminBench tau2-Bench TheAgentCompany AgentClinic MedChain
Builds on: Agent Skills MCP tau-bench HealthAdminBench
Closest papers by meaning
- PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (NeurIPS 2026 · similarity 0.87)
- MedMemoryBench: Benchmarking Agent Memory in Personalized Healthcare (NeurIPS 2026 · similarity 0.86)
- Reinforcement Learning for Tool-Calling Agents in Fast Healthcare Interoperability Resources (FHIR) (ICML 2026 · similarity 0.85)
- MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks (NeurIPS 2025 · similarity 0.85)
- EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings (ICML 2026 · similarity 0.85)
- Why Specialist Models Still Matter: A Heterogeneous Multi-Agent Paradigm for Medical Artificial Intelligence (ICML 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.