Research map / Large language models / Language, safety and interpretability
MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MedAlign, a clinician-authored benchmark of 983 instructions paired with longitudinal EHRs (303 with clinician gold responses), and evaluates six general LLMs with clinician correctness/ranking plus automated NLG metric correlations.
Problem
Existing EHR question answering datasets do not capture the complexity of clinician information needs or instruction-following over longitudinal, structured-plus-unstructured EHR data, and clinician review of LLM outputs is costly to repeat.
Approach
Decouple instruction collection from patient data: clinicians submit instructions without viewing EHRs, instructions are matched to EHRs via BM25 retrieval, and clinicians then write gold responses and rank LLM outputs; automated metrics (COMET, BERTScore) are checked for correlation with clinician rankings.
Key results
- GPT-4 (32k + multi-step refinement) achieved the highest clinician-judged correctness at 65.0%, while MPT-7B-Instruct reached 31.7%, so roughly 35% to 68% of responses were incorrect.
- Moving GPT-4 from 2k to 32k context raised clinician-judged correctness from 51.8% to 60.1%.
- COMET showed the strongest correlation with clinician rankings (Kendall's Tau 0.37), approaching inter-rater reliability of 0.44.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase GPT-4 context length from 2k to 32k (ablation) | GPT-4 · MedAlign | correctness: 51.8% -> 60.1%; 8.3% | Helped |
| add multi-step refinement to GPT-4 with 32k context (ablation) | GPT-4 · MedAlign | correctness: 60.1% -> 65.0% | Helped |
| increase Vicuña model size from 7B to 13B (ablation) | Vicuña · Calculation and scoring tasks | correctness: almost 20% | Helped |
| increase Vicuña model size from 7B to 13B (ablation) | Vicuña · Retrieve & Summarize and Diagnosis Support tasks | correctness | No clear effect |
| increase Vicuña model size from 7B to 13B (ablation) | Vicuña · Care Planning tasks | correctness | Hurt |
| match instructions to the top 5 EHRs using BM25 rather than uniform random sampling | Instruction-EHR matching · MedAlign | relevant-match rate: 38% -> 74% | Helped |
| exclude instructions that triggered content-filtering errors | GPT-4 (32k) evaluation · MedAlign, 259 instructions | correctness: 60.1% -> 69.9%; 9.8% | Helped |
| exclude instructions that triggered content-filtering errors | GPT-4 (MR) evaluation · MedAlign, 259 instructions | correctness: 65.0% -> 68.3%; 3.3% | Helped |
| exclude instructions that triggered content-filtering errors | GPT-4 (2k) evaluation · MedAlign, 259 instructions | correctness: 51.8% -> 52.5% | Helped |
Surprising findings
- Medical instruction-tuned models (AlpaCare, ClinicalCamel, Med42) mostly underperformed the base Llama 2 Chat model on MedAlign at matched size.
- GPT-4 returned content-filter errors for 44 of 303 instructions, likely triggered by anatomical descriptions in EHRs.
Limitations
- BM25 matching produced a relevant instruction-EHR pair for only about 74% of instructions, and about 30% had no relevant match.
- Clinicians sampled came from one academic medical center and cover only a small fraction of the clinician workforce.
- Hallucination failure modes were not measured.
Experimental setup
Datasets: MedAlign
Benchmarks: MedAlign
Metrics: correctness rate win rate mean human rank Kendall's Tau
Baselines: Vicuna-13B Vicuna-7B MPT-7B-Instruct AlpaCare ClinicalCamel Med42
Builds on: BM25 OMOP CDM COMET UniEval
Closest papers by meaning
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.90)
- EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries (NeurIPS 2024 · similarity 0.89)
- Instruction Tuning Large Language Models to Understand Electronic Health Records (NeurIPS 2024 · similarity 0.88)
- EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries (NeurIPS 2026 · similarity 0.88)
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.87)
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.