Research map / Large language models / Retrieval and knowledge
MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MedJourney, a Chinese benchmark of 12 datasets organized across four stages of the patient clinical journey, and evaluates proprietary, open, and medical LLMs in zero-shot and one-shot settings.
Problem
Existing medical LLM benchmarks are organized by question or task type (exams, QA, NER) rather than by the stages of a patient's real clinical journey, making it hard to assess LLM support for patients in practice.
Approach
Organize the benchmark around four patient journey stages (planning, access, delivery, ongoing care), with 12 tasks, 5 newly built by clinicians and 7 rebuilt from existing datasets, and evaluate with accuracy, NLG, entity-recall, and GPT-4 and human judgments.
Key results
- GPT-4 achieves the highest human-rated average (80) among 10 LLMs on the seven QA tasks, followed by QWen-32B (76).
- Zero-shot results show no single LLM dominates; HuatuoGPT2-34B leads the delivery-stage tasks (e.g., EP 0.757, DP 0.766, MP 0.777).
- Combining GPT-judge and entity recall yields 91.1% pairwise alignment with human rankings, versus 35.5% for BLEU-4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use one-shot prompting with ChatGPT | ChatGPT · Pre-Consultation Dialogue Summary | BLEU-4: 11.242 -> 21.316 | Helped |
| Use one-shot prompting with ChatGLM3 | ChatGLM3 · Patient-Doctor Dialogue Summarization | BLEU-4: 6.818 -> 20.674 | Helped |
| Use one-shot prompting with QWen-72B | QWen-72B · Disease Prediction | accuracy: 0.692 -> 0.769 | Helped |
| Use one-shot prompting with QWen-72B | QWen-72B · Treatment Prediction | accuracy: 0.635 -> 0.730 | Helped |
| Use one-shot prompting with HuatuoGPT2-34B | HuatuoGPT2-34B · Examination Prediction | accuracy: 0.757 -> 0.641 | Hurt |
| Use one-shot prompting with HuatuoGPT2-34B | HuatuoGPT2-34B · Disease Prediction | accuracy: 0.766 -> 0.588 | Hurt |
| Use one-shot prompting with HuatuoGPT2-34B | HuatuoGPT2-34B · Treatment Prediction | accuracy: 0.662 -> 0.466 | Hurt |
| Use one-shot prompting with HuatuoGPT2-34B | HuatuoGPT2-34B · Medication Prediction | accuracy: 0.777 -> 0.659 | Hurt |
| Use one-shot prompting | general models such as GPT-4 and QWen · MedJourney tasks | performance | Mixed |
Surprising findings
- BLEU-4 aligns poorly with human judgments (35.5% pairwise agreement), while the combined GPT-judge plus entity metric aligns best.
- Medical-specialized models such as DISC-MedLLM underperform general models, and one-shot examples degrade delivery-stage performance for medical-domain models.
Limitations
- Benchmark is Chinese-only, and English-centric models such as Llama were excluded.
- Scores are a reference only; high scores do not imply clinical readiness, and physician validation is needed.
- Several datasets are small (e.g., 100 pre-consultation dialogues, 133 hospital QA entries, 137 drug QA pairs).
Experimental setup
Datasets: MedDG IMCS21 CMB CMExam PsyQA Yixinli
Benchmarks: MedJourney
Metrics: accuracy BLEU-4 entity recall GPT-4 rating human rating
Baselines: ChatGLM3 QWen-7B QWen-14B QWen-32B QWen-72B ChatGPT GPT-4 HuatuoGPT2-7B HuatuoGPT2-34B DISC-MedLLM
Builds on: CBLUE PromptCBLUE MultiMedQA CMExam CMB
Closest papers by meaning
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.93)
- MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models (AAAI 2024 · similarity 0.92)
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations (NeurIPS 2024 · similarity 0.91)
- EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries (NeurIPS 2024 · similarity 0.91)
- Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity (ICML 2025 · similarity 0.91)
- Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.