Research map / Large language models / Agents, code and math
ClinicalLab: Aligning Agents for Multi-Departmental Clinical Diagnostics in the Real World
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ClinicalLab, comprising ClinicalBench (1,500 private real-case, end-to-end multi-department diagnosis benchmark), four ClinicalMetrics, and ClinicalAgent, a multi-agent clinical workflow that assigns different LLMs to departmental clinician roles.
Problem
Existing medical LLM benchmarks suffer from data leakage, neglect multi-departmental specialization, rely on multiple-choice questions, and do not evaluate end-to-end clinical diagnosis workflows.
Approach
A private, real-case, multi-department benchmark with generative end-to-end tasks (department guide, six-step clinical diagnosis, imaging diagnosis) plus a multi-agent ClinicalAgent that schedules departments and clinicians, with each clinician role powered by the LLM that performed best in that domain.
Key results
- InternLM2-20B-Chat achieved the top Avg. score of 51.98 and DWR of 14.91 among 17 evaluated LLMs on ClinicalBench.
- GPT-4 achieved the highest CDR of 33.27%, meaning it completed about a third of end-to-end consultations correctly.
- ClinicalAgent variant Agent#1@3 reached CDR 54.00 and Acceptability 18.22, versus 33.27 and 11.37 for GPT-4, and the highest human score (62.76) versus 58.84 for Gemini-Pro.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ClinicalAgent with 3 scheduled departments and 1 clinician per department instead of InternLM2 alone | ClinicalAgent on ClinicalBench · ClinicalBench | Avg.: 51.98 -> 54.46 | Helped |
| Use ClinicalAgent with 3 scheduled departments and 1 clinician per department instead of GPT-4 alone | ClinicalAgent on ClinicalBench · ClinicalBench | CDR: 33.27 -> 52.73 | Helped |
| Use ClinicalAgent with 3 scheduled departments and 1 clinician per department instead of GPT-4 alone | ClinicalAgent on ClinicalBench · ClinicalBench | Acceptability: 11.37 -> 17.82 | Helped |
| Use ClinicalAgent with 1 department and 3 clinicians from that department rather than 3 departments with 1 clinician each (ablation) | ClinicalAgent · ClinicalBench | CDR: 52.73 -> 54.00 | Helped |
| Use ClinicalAgent with 1 department and 3 clinicians from that department rather than 3 departments with 1 clinician each (ablation) | ClinicalAgent · ClinicalBench | Acceptability: 17.82 -> 18.22 | Helped |
| Use 1 scheduled department and 3 clinicians from it rather than 3 departments with 1 clinician each (ablation) | ClinicalAgent · ClinicalBench | Avg.: 54.46 -> 51.66 | Hurt |
| Use 1 scheduled department and 1 clinician rather than 3 departments with 1 clinician each (ablation) | ClinicalAgent · ClinicalBench | DWR: 14.30 -> 17.00 | Helped |
| Use 1 scheduled department and 1 clinician rather than 3 departments with 1 clinician each (ablation) | ClinicalAgent · ClinicalBench | CDR: 52.73 -> 35.13 | Hurt |
| Use 1 scheduled department and 1 clinician rather than 3 departments with 1 clinician each (ablation) | ClinicalAgent · ClinicalBench | Avg.: 54.46 -> 53.02 | Hurt |
Surprising findings
- GPT-4 scored highest under GPT-4o judging (90.93), likely due to judge self-preference, while human and automatic metrics favored ClinicalAgent.
- Most models showed near-perfect DIFR-Q but poor DIFR-N; GPT-4 had only 74.03% department name following.
Limitations
- Data come only from mainland Chinese Grade 3A hospitals and follow Chinese diagnostic procedures, limiting representativeness.
- ClinicalAgent was not compared to existing medical agents (MedAgent, Agent Hospital) due to incompatibility with generative evaluation.
- Few-shot learning and RAG were not explored; the metric trends across automatic, human, and GPT-4o evaluations do not fully align.
Experimental setup
Datasets: ClinicalBench
Benchmarks: ClinicalBench
Metrics: accuracy BLEU ROUGE BertScore DWR DIFR CDR Acceptability
Baselines: GPT-4 GPT-3.5 Gemini-Pro InternLM2-20B-Chat Yi-34B-Chat Qwen-72B-Chat Claude3 HuatuoGPT2-34B
Builds on: MedAgent Agent Hospital MDAgents
Closest papers by meaning
- PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (NeurIPS 2026 · similarity 0.88)
- MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks (NeurIPS 2025 · similarity 0.87)
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.86)
- MedChain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence (NeurIPS 2025 · similarity 0.86)
- Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.86)
- MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.