Research map / Multi-agent systems and games / LLM agents
MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MedAgentBoard, a benchmark comparing multi-agent LLM collaboration, single LLMs, and conventional methods across four medical task categories (QA/VQA, lay summarization, EHR prediction, clinical workflow automation), finding multi-agent methods do not consistently beat strong single LLMs or specialized conventional models.
Problem
Multi-agent LLM collaboration is claimed to help medical tasks, but prior evaluations are narrow in task coverage and rarely compare against both single LLMs and established conventional methods.
Approach
A unified benchmark spanning four diverse medical task categories and several data modalities, benchmarking multi-agent frameworks against single-LLM prompting and fine-tuned conventional models under the same pipeline.
Key results
- On MedQA multiple-choice, CoT-SC single LLM reaches 89.90% accuracy, while MedAgents reaches 85.25%; specialized VQA model MUMC reaches 91.40% on PathVQA vs. best multi-agent 75.90%.
- On lay summary (Cochrane), BART-CNN achieves 42.24% ROUGE-L versus 35.58% for the best single-LLM method and 34.25% for AgentSimp.
- On MIMIC-IV mortality EHR prediction, AdaCare reaches 94.28% AUROC versus 82.91% for ColaCare and 85.99% for GPT-4o; in clinical workflow automation, OpenManus achieves 64.0% correct on TJH modeling vs. 8.42% for single LLM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CoT-SC instead of zero-shot prompting (ablation) | DeepSeek-V3-0324 · MedQA MC | accuracy: 77.50\stddev{2.57} -> 89.90\stddev{2.43} | Helped |
| use CoT-SC instead of zero-shot prompting (ablation) | DeepSeek-V3-0324 · PubMedQA FF | LLM-as-a-judge score: 91.23\stddev{0.78} -> 84.25\stddev{1.14} | Hurt |
| use multi-agent MedAgents instead of the strongest single-LLM setting | DeepSeek-V3-0324 · MedQA MC | accuracy: 89.90\stddev{2.43} -> 85.25\stddev{2.67} | Hurt |
| use specialized conventional VLMs instead of general-purpose VLMs | medical VQA systems · PathVQA MC | accuracy: 74.50\stddev{3.20} -> 91.40\stddev{1.74} | Helped |
| add two in-context examples to optimized prompting (ablation) | DeepSeek-V3-0324 · Cochrane | ROUGE-L: 33.85\stddev{0.60} -> 35.58\stddev{0.63} | Helped |
| add two in-context examples to optimized prompting (ablation) | DeepSeek-V3-0324 · Cochrane | SARI: 38.25\stddev{0.41} -> 38.56\stddev{0.28} | Helped |
| use AgentSimp instead of the best single LLM prompting setting | DeepSeek-V3-0324 · Cochrane | ROUGE-L: 35.58\stddev{0.63} -> 34.25\stddev{0.55} | Hurt |
| use fine-tuned conventional models instead of LLM-based methods | lay summary generation · PLABA | ROUGE-L: 41.37\stddev{0.42} -> 59.11\stddev{1.02} | Helped |
| use conventional AdaCare instead of zero-shot GPT-4o | MIMIC-IV mortality prediction · MIMIC-IV mortality | AUROC: 85.99\stddev{3.85} -> 94.28\stddev{3.52} | Helped |
| use multi-agent ColaCare instead of its single base LLM | DeepSeek-V3-0324 · MIMIC-IV mortality | AUROC: 76.86\stddev{4.71} -> 82.91\stddev{4.49} | Helped |
| use conventional AdaCare instead of multi-agent ColaCare | MIMIC-IV mortality prediction · MIMIC-IV mortality | AUROC: 82.91\stddev{4.49} -> 94.28\stddev{3.52} | Helped |
| use multi-agent workflow automation instead of a single LLM | clinical workflow automation · TJH modeling workflow | Correct: 8.42\% -> 64.0\% | Helped |
Surprising findings
- Advanced single LLMs (e.g., DeepSeek-V3 zero-shot) can match or exceed multi-agent collaboration on textual QA and lay summaries.
- Multi-agent gains on EHR prediction appear largely driven by the strength of the underlying base LLM rather than the collaboration mechanism.
Limitations
- Results depend on underlying foundation models and may shift as LLMs improve.
- Clinical workflow automation uses only a curated set of 100 tasks, not exhaustive.
- Evaluation relies largely on automated metrics and limited human assessment.
Experimental setup
Datasets: MedQA PubMedQA PathVQA VQA-RAD Cochrane eLife PLOS Med-EASi PLABA MIMIC-IV TJH
Benchmarks: MedQA PubMedQA PathVQA VQA-RAD Cochrane eLife PLOS Med-EASi PLABA MIMIC-IV TJH
Metrics: accuracy LLM-as-a-judge score ROUGE-L SARI AUROC AUPRC correct-rate by expert panel
Baselines: BioLinkBERT Gatortron M³AE BiomedGPT MUMC LLaVA-Med Med-Flamingo BART T5 PEGASUS BART-CNN XGBoost
Builds on: MedAgents ReConcile MDAgents ColaCare AgentSimp
Closest papers by meaning
- MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making (NeurIPS 2024 · similarity 0.92)
- MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow (ICLR 2026 · similarity 0.89)
- PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (NeurIPS 2026 · similarity 0.88)
- MedChain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence (NeurIPS 2025 · similarity 0.88)
- MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoning (ICLR 2026 · similarity 0.88)
- MedExAgent: Training LLM Agents to Ask, Examine, and Diagnose in Noisy Clinical Environments (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.