Research map / Large language models / Instruction and fine-tuning
Instruction Tuning Large Language Models to Understand Electronic Health Records
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MIMIC-Instr, a 400K instruction-following dataset built from MIMIC-IV EHRs, and Llemr, an LLM framework that encodes event sequences with ClinicalBERT and a linear projection into a Vicuna-7B backbone, trained in two curriculum stages.
Problem
Building a conversational AI assistant for structured EHR data is hard because large-scale instruction-following data grounded in EHRs is lacking and existing model architectures struggle with heterogeneous, complex EHR schemas.
Approach
Represent each patient's EHR as a stream of timestamp-type-value events encoded by ClinicalBERT, map them into an LLM's embedding space with a linear projection, and tune in two curriculum stages: schema alignment on template-based QA, then clinical reasoning on discharge-summary-based QA.
Key results
- Llemr + Stage 1&2 reaches 73.33 relative score overall on the conversational test set, above all seven open LLM baselines (best baseline dolphin-2.0-mistral-7b at 67.86).
- On MIMIC-IV clinical predictive tasks, fine-tuned Llemr achieves 0.8388 AUC-ROC on mortality and 0.7251 on readmission, performing better than or on par with baselines like REMed and GRASP.
- Llemr matches about 73% of GPT-4 reference performance on the instruction-following evaluation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Stage 1 schema alignment training (ablation) | Llemr · Schema Alignment | relative score: 66.81 ± 5.61 -> 69.71 ± 6.32 | Helped |
| Add Stage 2 clinical reasoning training after Stage 1 (ablation) | Llemr · Clinical Reasoning | relative score: 64.35 ± 7.21 -> 76.23 ± 4.23 | Helped |
| Add Stage 2 clinical reasoning training after Stage 1 (ablation) | Llemr · Overall | relative score: 67.03 ± 6.83 -> 73.33 ± 5.30 | Helped |
| Use Llemr instead of the strongest listed baseline | MIMIC-IV mortality prediction · MIMIC-IV mortality prediction | AUC-ROC: 0.8362 (0.01) -> 0.8388 (0.01) | Helped |
| Use Llemr instead of the strongest listed baseline | MIMIC-IV readmission prediction · MIMIC-IV readmission prediction | AUC-ROC: 0.7193 (0.02) -> 0.7251 (0.03) | Helped |
| Use Llemr instead of the strongest listed baseline | MIMIC-IV length-of-stay prediction · MIMIC-IV length-of-stay prediction | AUC-ROC: 0.7100 (0.03) -> 0.7132 (0.01) | Helped |
| Use Llemr instead of the strongest listed baseline | MIMIC-IV diagnosis classification · MIMIC-IV diagnosis classification | macro AUC-ROC: 0.8103 (0.03) -> 0.8086 (0.01) | Hurt |
Surprising findings
- Stage-1 training that updates only the linear projection layer alone improves over the vicuna-7b-v1.5 backbone (67.03 vs 64.60 overall).
- Clinical reasoning data generated from discharge summaries can include information absent from EHR tables, leading to answers unsupported by the event sequence.
Limitations
- Clinical reasoning answers can draw on discharge summaries rather than the structured EHR, so some answers are not grounded in the input events.
- Evaluation of the conversational assistant relies on GPT-based judging and a small 200-question test set.
- Clinical predictive results require an additional supervised fine-tuning step and only comparable to SOTA rather than clearly superior.
Experimental setup
Datasets: MIMIC-IV MIMIC-Instr
Benchmarks: MIMIC-Instr conversational test set (200 QA) MIMIC-IV clinical predictive tasks (mortality, readmission, length-of-stay, diagnosis)
Metrics: relative GPT score vs GPT-4 reference AUC-ROC macro AUC-ROC
Baselines: Llama-2-7b-chat-hf SynthIA-7B-v1.3 Mistral-7B-OpenOrca Llama-3-8b-Instruct MPT-7b-8k-instruct vicuna-7b-v1.5 dolphin-2.0-mistral-7b RNN Transformer RETAIN GRASP GenHPF
Builds on: LLaVA REMed Medical Event Data Standard
Closest papers by meaning
- Toward Better EHR Reasoning in LLMs: Reinforcement Learning with Expert Attention Guidance (AAAI 2026 · similarity 0.89)
- MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records (AAAI 2024 · similarity 0.88)
- From Conversation to Query Execution: Benchmarking User and Tool Interactions for EHR Database Agents (ICLR 2026 · similarity 0.87)
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.87)
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.86)
- Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-World Multi-Turn Dialogue (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.