Research map / Large language models / Agents, code and math
ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ER-Reason, a benchmark of 25K de-identified ER clinical notes with acuity, diagnosis, disposition tasks and physician-annotated SCT-style sequential belief-updating evaluation, testing eight LLMs.
Problem
Existing LLM clinical reasoning benchmarks conflate reasoning with knowledge recall and diagnostic accuracy and rely on stylized vignettes rather than real clinical documentation, so they do not measure how models revise beliefs as evidence accumulates across the emergency workflow.
Approach
Operationalizes clinical reasoning as sequential belief updating: evaluates LLMs on real ER notes across triage, treatment, disposition and diagnosis, plus SCT-style stepwise items scored against physician consensus with new metrics (DxUpdate, DxTrajectory, FinalDx) and cross-stage coherence/workflow accuracy.
Key results
- Best zero-shot final diagnosis ICD-10 exact match is 39.91% (GPT-5.2 Thinking); CCSR accuracy reaches 45.09%.
- DxUpdate and DxTrajectory improve substantially from zero-shot to single/full oracle, e.g., o4-mini DxUpdate 0.691 ZS vs 0.809 SO.
- GPT-5.2 Thinking shows no significant difference from GPT-5.2 on DxUpdate (p=0.455) or DxTrajectory (p=0.192), so reasoning models do not outperform non-reasoning ones on sequential SCT tasks.
Surprising findings
- Reasoning models do not outperform non-reasoning models on sequential SCT-style belief updating; Claude 4.5 Thinking significantly underperforms its base model on DxTrajectory.
- Strong per-task performance can mask cross-stage failures: Gemini 2.5 Flash has second-highest high-acuity workflow accuracy but one of the lowest low-acuity scores.
Limitations
- Does not account for hospital-level contextual factors such as bed availability, staffing, and institutional protocols.
- Did not control or vary the inference-time thinking budget for reasoning models.
- Single academic medical center data; SCT annotated by only three emergency physicians rather than a large panel.
Experimental setup
Datasets: ER-Reason clinical notes (25,174 notes, 3,437 patients) SCT physician annotations (2,555 annotations over 194 cases)
Benchmarks: ER-Reason clinical decision tasks ER-Reason SCT reasoning tasks ER-Reason cross-stage workflow analysis
Metrics: accuracy linearly weighted Cohen's kappa (DxUpdate) Spearman rho (DxTrajectory, FinalDx) intra-case coherence CCSR category accuracy
Baselines: DeepSeek-R1 o4-mini Gemini 2.5 Flash GPT-5.2 Thinking Claude Sonnet 4.5 Thinking Phi-4 GPT-5.2 Claude Sonnet 4.5
Builds on: Script Concordance Test Ledley and Lusted diagnostic model step-back prompting McCoy et al. SCT benchmark
Closest papers by meaning
- DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Models (NeurIPS 2024 · similarity 0.89)
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.88)
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations (NeurIPS 2024 · similarity 0.88)
- EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries (NeurIPS 2024 · similarity 0.88)
- Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.88)
- EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.