Research map / Multi-agent systems and games / LLM agents

Why Specialist Models Still Matter: A Heterogeneous Multi-Agent Paradigm for Medical Artificial Intelligence

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agent data synthesis and behavior studies · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HetMedAgent, a heterogeneous multi-agent framework coordinating GPT-4o-based generalist LLM agents, in-house Transformer specialist models for ECHO and ECG, and clinician escalation via uncertainty-based routing, evaluated on a private cardiovascular cohort.

Problem

Whether domain-specific medical specialist models remain valuable alongside generalist LLMs, and how to combine generalist LLMs, specialist models and clinicians for multimodal clinical decision-making with safe escalation to clinicians.

Approach

A heterogeneous multi-agent architecture where specialist models convert multimodal data into text findings with generation confidence, a generalist LLM fuses them with conflict-aware weights, and a composite uncertainty score (confidence, cross-specialist conflict, reasoning coherence) routes high-uncertainty cases to a clinician agent whose feedback adaptively calibrates the threshold.

Key results

• HetMedAgent (w/o clinician, GPT-4o) reaches 0.866 AUROC / 0.844 F1 on risk stratification, outperforming the best medical LLM (Meditron, 0.801/0.768) and best multi-agent baseline (MedAgents, 0.823/0.789).
• Averaged over three tasks, the full system with both specialists reaches 0.798 AUROC and 0.773 F1 versus 0.671 / 0.625 for GPT-4o alone.
• Adaptive threshold calibration improves average AIR from 1.468 to 1.679 and reduces intervention cases from 18.6% to 15.8%.

Surprising findings

• Claude-3.5-Sonnet, Gemini and open-weight Llama-3.3-70B and Qwen-2.5-72B perform only slightly below GPT-4o within the framework, suggesting the specialist inputs dominate performance.
• Baseline medical LLMs such as Meditron-70B score below general-purpose GPT-4o-based systems on the cardiovascular tasks, and the paper's table indicates baselines were given the same multimodal inputs.

Limitations

• Evaluation is single-institution on a private retrospective cohort of 613 cardiovascular cases; clinician feedback in threshold calibration is simulated with ground truth.
• Generation confidence reflects token-level certainty rather than clinical correctness, and routing ignores clinical severity.
• Relies on commercial LLM APIs (GPT-4o), raising privacy concerns; inter-agent interface is text-only.

Experimental setup

Datasets: Private cardiovascular cohort (ECHO reports, ECG images; 613 test cases from 514 patients) IU X-Ray (cross-domain validation, 668 test cases)

Benchmarks: Private cardiovascular test set (admission risk stratification, etiology prediction, severity assessment) IU X-Ray acute vs non-acute

Metrics: AUROC F1 BERTScore conflict score AIR

Baselines: PMC-LLaMA Meditron BioMistral Llama-3-Meditron DoctorGLM GatorTron MedAgents AgentClinic AutoGen MetaGPT

Builds on: AutoGen MetaGPT AgentClinic MedAgents

Closest papers by meaning

• MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making (NeurIPS 2024 · similarity 0.90)
• MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoning (ICLR 2026 · similarity 0.88)
• MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks (NeurIPS 2025 · similarity 0.87)
• Doctor-R1: Mastering Clinical Inquiry with Experiential Agentic Reinforcement Learning (ICLR 2026 · similarity 0.87)
• MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow (ICLR 2026 · similarity 0.86)
• MedChain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence (NeurIPS 2025 · similarity 0.86)