Research map / Multi-agent systems and games / LLM agents
MedChain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MedChain, a 12,163-case Chinese clinical workflow benchmark with five sequential interactive stages, and MedChain-Agent, a multi-agent system with feedback and a case-retrieval RAG module that outperforms baselines.
Problem
LLM agents are evaluated on medical licensing-style QA, which misses personalization, interactivity, and sequential dependencies of real clinical decision-making across the full care workflow.
Approach
A sequential, interactive clinical benchmark (specialty referral, history-taking with a simulated patient, examination/report generation, diagnosis, treatment) paired with a multi-agent framework that adds cross-stage feedback and a MedCase-RAG retrieving similar cases via 12-dimensional structured case representations.
Key results
- MedChain-Agent (InternVL2-8B backbone) reaches average 0.5200 on MedChain, versus 0.4262 for the best base LLM (Claude-3.5-sonnet) and 0.4704 for single-agent RAG.
- Applying the framework lifts InternVL2 average by 13.41 points and GPT-4o-mini by 8.70 points on MedChain.
- On MedQA, PubMedQA, PathVQA and MedBullets, MedChain-Agent achieves the highest average (0.576) among the compared agent methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Feedback to MedChain-Agent without Feedback or MedCase-RAG (ablation) | MedChain-Agent on MedChain · MedChain | Average: 0.4341 -> 0.4692 | Helped |
| add MedCase-RAG to MedChain-Agent without Feedback or MedCase-RAG (ablation) | MedChain-Agent on MedChain · MedChain | Average: 0.4341 -> 0.5140 | Helped |
| add both Feedback and MedCase-RAG (ablation) | MedChain-Agent on MedChain · MedChain | Average: 0.4341 -> 0.5200 | Helped |
| add detailed patient profiles (ablation) | MedChain-Agent · MedChain Diagnosis | Score: 0.4159 -> 0.4802 | Helped |
| add detailed patient profiles (ablation) | MedChain-Agent · MedChain Treatment | Score: 0.4310 -> 0.4613 | Helped |
| retain sequential dependency between stages (ablation) | MedChain-Agent · MedChain Diagnosis | Score: 0.4807 -> 0.4802 | Hurt |
| retain sequential dependency between stages (ablation) | MedChain-Agent · MedChain Treatment | Score: 0.4743 -> 0.4613 | Hurt |
| require interactive information gathering (ablation) | MedChain-Agent · MedChain Diagnosis | Score: 0.4634 -> 0.4802 | Helped |
| require interactive information gathering (ablation) | MedChain-Agent · MedChain Treatment | Score: 0.5207 -> 0.4613 | Hurt |
| apply MedChain-Agent instead of MDAgents | InternVL2-based agents · Other diagnosis benchmarks | Average: 0.546 -> 0.576 | Helped |
Surprising findings
- Removing sequential dependency (using ground-truth previous-stage outputs) improved most models' diagnosis scores, indicating error propagation makes the task harder.
- Multi-agent baselines such as MDAgent and MedAgent often underperform single-agent RAG and even base models on MedChain.
Limitations
- Data comes from a single Chinese website (iiYi) with imbalanced specialty distribution.
- Patient responses during history-taking are simulated by Gemma2-9B, which may not reflect real patient variability.
- Dataset and code are promised only upon acceptance, not released with the submission.
Experimental setup
Datasets: iiYi clinical cases (12,163 EHRs) MedQA PubMedQA PathVQA MedBullets
Benchmarks: MedChain MedQA PubMedQA PathVQA MedBullets
Metrics: accuracy IoU claim recall five-level diagnosis score average score
Baselines: GPT-4o-mini Claude-3.5-sonnet MedGemma LLaVA Qwen2 InternVL2 Baichuan HuaTuoGPT MedReason FineMedLM-o1 CoT Self-consistency
Builds on: MedAgents MDAgents AutoGen DyLAN DocLens
Closest papers by meaning
- MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making (NeurIPS 2024 · similarity 0.89)
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.89)
- MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks (NeurIPS 2025 · similarity 0.88)
- MedExAgent: Training LLM Agents to Ask, Examine, and Diagnose in Noisy Clinical Environments (NeurIPS 2026 · similarity 0.88)
- MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncology (NeurIPS 2025 · similarity 0.88)
- PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.