Research map / Large language models / Language, safety and interpretability
Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHR Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically evaluates context length (512 to 16k tokens) for GPT, Llama, Mamba and Hyena EHR foundation models pretrained on 2.5M patients, showing Mamba at 16k beats prior SOTA on EHRSHOT and that EHR-specific properties degrade performance but longer contexts are more robust.
Problem
Existing EHR foundation models use context windows under 1k tokens, so they cannot model full longitudinal patient records that can exceed tens of thousands of events; the effect of long context and subquadratic architectures on EHR clinical prediction is unexplored.
Approach
Pretrain 120M-scale GPT, Llama, Mamba and Hyena models over multiple context lengths on longitudinal structured EHRs, then stratify downstream performance by three EHR-specific properties (copy-forwarding repetition, irregular inter-event intervals, disease progression perplexity) to test robustness.
Key results
- Mamba at 16k context reaches mean AUROC 0.807 across 14 EHRSHOT tasks, +0.03 over prior SOTA CLMBR-t-base (0.777), beating SOTA on 9/14 tasks.
- Higher irregularity correlates with worse performance: about 14% higher Brier score for the most irregular versus least irregular patients (CLMBR-t-base stratification).
- Hyena degrades sharply beyond 4k context, while longer Mamba and Llama contexts achieve lower perplexity at later token positions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase Mamba context length to 16k tokens | Mamba · EHRSHOT (14 tasks) | mean AUROC: +0.03 points over prior SOTA | Helped |
| Increase context length (ablation) | Mamba and Llama · EHRSHOT clinical prediction tasks | AUROC | Helped |
| Increase context length beyond 4k tokens (ablation) | Hyena · EHRSHOT clinical prediction tasks | AUROC | Hurt |
| Use Artificial Time Tokens (ATT) in tokenization (ablation) | EHR prediction models · EHRSHOT clinical prediction tasks | AUROC | Hurt |
| Increase patient timeline repetitiveness | EHR prediction models · EHRSHOT, repetitiveness quartiles | Brier score | Hurt |
| Increase context from 1k to 16k tokens (ablation) | Mamba · EHRSHOT, repetitiveness quartiles | Brier score: 0.0644 -> 0.0605; 0.0737 -> 0.0670; 0.0744 -> 0.0700; 0.0790 -> 0.0746 | Helped |
| Increase patient timeline irregularity | EHR prediction models · EHRSHOT, irregularity quartiles | Brier score | Hurt |
| Increase context from 1k to 16k tokens (ablation) | Mamba · EHRSHOT, irregularity quartiles | Brier score: 0.0693 -> 0.0641; 0.0729 -> 0.0678; 0.0731 -> 0.0679; 0.0764 -> 0.0723 | Helped |
| Evaluate patients with the most irregular rather than least irregular timelines | EHR prediction models · EHRSHOT | Brier score: 14% worse | Hurt |
| Use longer context (ablation) | Mamba and Llama · EHR-OMOP token prediction | perplexity | Helped |
| Use longer context (ablation) | Hyena · EHR-OMOP token prediction | perplexity | Hurt |
| Use longer contexts of 2k and 4k tokens (ablation) | GPT · EHR-OMOP token prediction | perplexity | Mixed |
Surprising findings
- Later tokens in EHR sequences have higher perplexity even when conditioning on all prior tokens, contrary to the natural-language trend.
- Hyena's performance drops sharply after 4k tokens, so longer context does not uniformly help.
Limitations
- Transformers (GPT, Llama) were not evaluated beyond 4k tokens due to compute limits.
- Pretraining data come from a single institution, limiting generalizability.
- Only three EHR-specific properties and clinical risk prediction tasks were studied.
Experimental setup
Datasets: EHR-OMOP (2.5M train, 0.5M validation patients from an academic medical center) EHRSHOT WikiText-103
Benchmarks: EHRSHOT (14 binary tasks)
Metrics: AUROC Brier score perplexity
Baselines: CLMBR-t-base EHRMamba GPT Hyena
Builds on: Mamba EHRSHOT CLMBR CEHR-BERT Hyena
Closest papers by meaning
- One Loss to Rule Them All: Marked Time-to-Event for Structured EHR Foundation Models (NeurIPS 2026 · similarity 0.91)
- Risk Horizons: Structured Hypothesis Spaces for Longitudinal Clinical Prediction (NeurIPS 2026 · similarity 0.88)
- MOTOR: A Time-to-Event Foundation Model For Structured Medical Records (ICLR 2024 · similarity 0.86)
- EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries (NeurIPS 2024 · similarity 0.86)
- Instruction Tuning Large Language Models to Understand Electronic Health Records (NeurIPS 2024 · similarity 0.86)
- Toward Better EHR Reasoning in LLMs: Reinforcement Learning with Expert Attention Guidance (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.