Research map / Multi-agent systems and games / LLM agents

PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PhysicianBench, a physician-validated benchmark of 100 long-horizon EHR tasks in a FHIR sandbox, graded by 670 checkpoints, and evaluates 12 LLM agents (best: GPT-5.5 at 46% pass@1).

Problem

Existing medical agent benchmarks test static knowledge, single-step actions, or action intent without verifying execution in an EHR, so they miss long-horizon clinical workflows.

Approach

Tasks adapted from real e-consult cases are instantiated in per-task HAPI FHIR servers with perturbed real patient records; agents act through FHIR tools, and checkpoints combine code, hybrid, and LLM-judge graders over the post-execution EHR state, with physician validation via an agentic-revision and diff-approval pipeline.

Key results

• GPT-5.5 achieves 46.3% pass@1 and 28.0% pass^3, the best of 12 agents; Claude Opus 4.6 reaches 31.7%.
• The best open-source model, DeepSeek V4-Pro, reaches 18.7% pass@1 (Kimi-K2.6 17.0%).
• Clinical reasoning accounts for roughly 50% of failed checkpoints on average (50.4%), with action execution at 18.8%.

Surprising findings

• Open-source DeepSeek V4-Pro (18.7%) outperforms several proprietary models such as Gemini Pro 3.1 and Grok-4.20.
• Documentation failures are more pronounced in stronger models, which attempt more complete notes and are penalized for omissions.

Limitations

• Scope limited to e-consult-style, EHR-mediated physician work; no discharge summaries or inpatient settings yet.
• Annotators act as second-tier reviewers of e-consult answers rather than de novo subspecialty authors.
• Patient records are perturbed/de-identified, which may reduce realism; LLM-judge grading adds grader dependence.

Experimental setup

Datasets: STARR de-identified e-consult cases (185 candidate, 100 tasks retained)

Benchmarks: PhysicianBench

Metrics: pass@1 pass@3 pass^3 average tool-calling turns

Baselines: Claude Opus 4.6 Claude Opus 4.7 GPT-5.4 Claude Sonnet 4.6 MiMo-v2.5-Pro Qwen3.6-Plus MiniMax M2.7 Gemini Pro 3.1 Grok-4.20 DeepSeek V4-Pro Kimi-K2.6

Builds on: MedAgentBench tau-bench HAPI FHIR DICTUM

Closest papers by meaning

• MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks (NeurIPS 2025 · similarity 0.88)
• MedMemoryBench: Benchmarking Agent Memory in Personalized Healthcare (NeurIPS 2026 · similarity 0.88)
• MedChain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence (NeurIPS 2025 · similarity 0.88)
• ClinicalLab: Aligning Agents for Multi-Departmental Clinical Diagnostics in the Real World (NeurIPS 2025 · similarity 0.88)
• MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making (NeurIPS 2024 · similarity 0.87)
• $\chi$-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows? (NeurIPS 2026 · similarity 0.87)