Research map / Large language models / Retrieval and knowledge

EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries

NeurIPS 2026 · Large language models › Retrieval and knowledge · approach: Domain-specific QA datasets · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces EHRNote-ChatQA, a 967-patient multi-turn, multi-note clinical QA benchmark over MIMIC-IV discharge summaries with paired evidence-grounding questions, and benchmarks 22 LLMs.

Problem

Existing clinical QA benchmarks are single-turn, exam-style, or lack per-answer evidence grounding across a patient's multiple discharge summaries.

Approach

Multi-turn, multi-note MCQ benchmark where each content question is paired with an evidence-grounding question asking for the minimal note/section sources, built via an expert-informed schema and templates with LLM generation (Gemini-2.5-Pro) and full expert review.

Key results

• gpt-5.4 achieves 98.33% QA-level content accuracy but only 89.02% source accuracy and 40.02% sample-level paired accuracy.
• Across 22 models, source accuracy is on average 17.56 pp lower than content accuracy.
• Average content accuracy drops from 78.3% after a correct prior turn to 62.6% after an incorrect one (gap 15.7 pp).

Surprising findings

• Medical-domain specialized models (MedGemma, MediPhi) do not consistently outperform similarly sized general-purpose models on multi-turn clinical QA.
• Strong single-turn EHRNoteQA performance (e.g., DeepSeek-R1-Distill-Qwen-7B at 80.15%) does not transfer; its multi-turn content accuracy is 27.05%.

Limitations

• Single hospital system (MIMIC-IV) data may not generalize across institutions.
• Only discharge summaries are covered, not other EHR note types or structured data.
• Each sample reviewed by a single expert rather than multiple annotators.

Experimental setup

Datasets: MIMIC-IV discharge summaries

Benchmarks: EHRNote-ChatQA EHRNoteQA

Metrics: accuracy content accuracy source accuracy paired accuracy sample-level 0/1 accuracy

Baselines: EHRNoteQA gpt-5.4 gpt-5.4-mini gemini-3-flash-preview Llama-4-Scout-17B-16E-Instruct Qwen3-Next-80B-A3B-Instruct Qwen3-30B-A3B-Instruct-2507 medgemma-27b-it

Builds on: EHRNoteQA MT-Bench ArchEHR-QA

Closest papers by meaning

• EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries (NeurIPS 2024 · similarity 0.95)
• MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.88)
• MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records (AAAI 2024 · similarity 0.88)
• ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room (NeurIPS 2026 · similarity 0.87)
• MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning (NeurIPS 2024 · similarity 0.87)
• CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcare (AAAI 2024 · similarity 0.87)