Research map / Large language models / Retrieval and knowledge
EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EHRNoteQA, a 962-question clinician-reviewed benchmark over MIMIC-IV discharge summaries (open-ended and multi-choice), evaluates 27 LLMs, and shows its scores correlate better with clinician judgments than other benchmarks.
Problem
Existing clinical QA benchmarks focus on general medical knowledge or single-note questions, failing to evaluate LLMs on clinician-style questions requiring information across multiple discharge summaries of the same patient.
Approach
Build QA pairs from accumulated MIMIC-IV discharge summaries (1-3 admissions per patient) by generating them with GPT-4 and then manually reviewing and revising each by three clinicians; evaluate both open-ended (GPT-4 judged) and multi-choice formats, and validate against clinician-scored LLM performance.
Key results
- GPT-4 scores highest on EHRNoteQA (multi-choice 97.16 and open-ended 91.30 at Level 1), with Llama3-70b-Instruct close behind (94.33 multi-choice, 89.04 open-ended).
- EHRNoteQA Level 1 scores correlate with clinician-evaluated LLM scores at Spearman 0.78 and Kendall 0.62 on average, outperforming other benchmarks such as emrQA and MedQA.
- GPT-4 judge agreement with clinicians (Cohen's kappa 0.757, 0.855, 0.880) and multi-choice GPT-4 evaluation showed 98% manual accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EHRNoteQA open-ended scores rather than emrQA scores to proxy clinician evaluation | LLM evaluation against clinician-evaluated performance · EHRNoteQA Level 1 vs emrQA | Spearman correlation: 0.770 vs 0.696 (Clinician A); 0.805 vs 0.653 (Clinician B); 0.801 vs 0.661 (Clinician C) | Helped |
| Use EHRNoteQA multi-choice scores rather than emrQA scores to proxy clinician evaluation | LLM evaluation against clinician-evaluated performance · EHRNoteQA Level 1 vs emrQA | Spearman correlation: 0.766 vs 0.696 (Clinician A); 0.732 vs 0.653 (Clinician B); 0.812 vs 0.661 (Clinician C) | Helped |
| Use GPT-4 evaluation instead of BLEU for open-ended answers | EHRNoteQA open-ended evaluation · EHRNoteQA Open-Ended | Spearman correlation: 0.770 vs 0.155 (Clinician A); 0.805 vs 0.037 (Clinician B); 0.801 vs 0.014 (Clinician C) | Helped |
| Use GPT-4 evaluation instead of probability(index) scoring | EHRNoteQA multi-choice evaluation · EHRNoteQA Multi-Choice | Spearman correlation: 0.766 vs 0.622 (Clinician A); 0.732 vs 0.596 (Clinician B); 0.812 vs 0.676 (Clinician C) | Helped |
| Use GPT-4 evaluation instead of probability(value) scoring | EHRNoteQA multi-choice evaluation · EHRNoteQA Multi-Choice | Spearman correlation: 0.766 vs 0.514 (Clinician A); 0.732 vs 0.523 (Clinician B); 0.812 vs 0.549 (Clinician C) | Helped |
| Evaluate longer Level 2 contexts instead of Level 1 contexts | models supporting at least 8k context · EHRNoteQA | accuracy | Hurt |
| Increase the number of notes from one to multiple notes | LLMs evaluated on EHRNoteQA · EHRNoteQA | accuracy | Hurt |
| Increase notes from one to two for Camel-Platypus2-13b | Camel-Platypus2-13b · EHRNoteQA Level 1 | accuracy: 5.21% reduction | Hurt |
| Increase notes from one to two for Llama2-13b-chat | Llama2-13b-chat · EHRNoteQA Level 1 | accuracy: 13.5% drop | Hurt |
| Use Mistral-7b rather than Llama2-7b as the foundation model | same-size instruction-tuned models · EHRNoteQA | average score: 73.29 vs 65.69 | Helped |
| Use GPT-4 rather than GPT-4-Turbo | LLM performance on EHRNoteQA · EHRNoteQA | accuracy: Multi-Choice Level 1: 97.16 vs 95.27; Level 2: 95.15 vs 94.23; Open-Ended Level 1: 91.30 vs 91.30; Level 2: 89.61 vs 86. | Mixed |
Surprising findings
- Mistral-7b-OpenOrca (7B) outperforms Llama2-70b-chat (70B) on EHRNoteQA.
- PubMedQA scores correlate very weakly with clinician-evaluated performance (Spearman 0.07 to 0.17).
Limitations
- Built only on MIMIC-IV from one institution (BIDMC), so results may not transfer to other EHR systems.
- Clinician-correlation experiment used questions derived from MIMIC-III (DiSCQ) rather than EHRNoteQA itself.
- Relies on GPT-4 for question/answer generation and for judging open-ended and multi-choice outputs.
Experimental setup
Datasets: MIMIC-IV discharge summaries DiSCQ MIMIC-III discharge summaries emrQA MedQA MedMCQA PubMedQA MMLU
Benchmarks: EHRNoteQA MedQA MedMCQA PubMedQA MMLU ARC HellaSwag TruthfulQA Winogrande GSM8K
Metrics: accuracy Spearman correlation Kendall correlation Cohen's kappa
Baselines: MedQA MedMCQA PubMedQA MMLU emrQA BLEU ROUGE-L SentenceBERT
Builds on: emrQA MIMIC-IV
Closest papers by meaning
- EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries (NeurIPS 2026 · similarity 0.95)
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.91)
- MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records (AAAI 2024 · similarity 0.89)
- ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room (NeurIPS 2026 · similarity 0.88)
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.87)
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.