Research map / Large language models / Retrieval and knowledge
EHRCon: Dataset for Checking Consistency between Unstructured Notes and Structured Tables in Electronic Health Records
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EHRCon, a dataset of 4,101 entities from 105 MIMIC-III clinical notes checked for consistency against structured tables (plus an OMOP CDM version), and CheckEHR, an eight-stage LLM pipeline for this check.
Problem
Inconsistencies between unstructured clinical notes and structured EHR tables pose patient safety risks, yet prior table-fact-verification work handles only single claims against small tables, not large relational EHR databases.
Approach
Frame consistency checking as a note-to-relational-database task with Consistent/Inconsistent labels, and decompose it into eight LLM stages (note segmentation, NER, time filtering, table identification, pseudo table creation, self-correction, value reformatting, query generation) that produce SQL queries.
Key results
- GPT-3.5 (0613) in the few-shot setting with CheckEHR reaches 61.06% recall, 48.63% precision and 86.25% intersection on MIMIC-III.
- Llama-3 70B zero-shot reaches 52.39% recall on MIMIC-III; on MIMIC-OMOP few-shot Llama-3 reaches 55.71% total recall and 65.07% precision.
- Providing ground-truth stages raises GPT-3.5 recall from 76.11% (NER given) to 92.83% (pseudo table, self-correction, and reformatting also given), suggesting LLMs struggle with converting free text to structured form.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the full CheckEHR framework instead of direct GPT-3.5 (0613) consistency checking | GPT-3.5 (0613) on MIMIC-III · MIMIC-III | recall: 10.12% -> 61.06% | Helped |
| Use the full CheckEHR framework instead of direct GPT-3.5 (0613) consistency checking | GPT-3.5 (0613) on MIMIC-III · MIMIC-III | precision: 8.75% -> 48.63% | Helped |
| Add few-shot examples | Tulu2 with CheckEHR · MIMIC-III, Total | recall: 12.08 -> 44.95 | Helped |
| Add few-shot examples | Mixtral with CheckEHR · MIMIC-III, Total | recall | Helped |
| Add few-shot examples | GPT-3.5 (0613) with CheckEHR · MIMIC-III, Total | recall: 42.83 -> 61.06 | Helped |
| Add few-shot examples | Llama-3 with CheckEHR · MIMIC-III, Total | recall: 52.39 -> 53.05 | No clear effect |
| Add few-shot examples | Llama-3 with CheckEHR · MIMIC-III, Nursing Note | recall: 53.45 -> 52.60 | Hurt |
| Provide gold time filtering and table identification, in addition to gold entities (ablation) | CheckEHR, GPT-3.5 (0613) component analysis · MIMIC-III, 25% test set | recall: 76.11% -> 82.49% | Helped |
| Provide gold pseudo-table creation, self-correction, and value reformatting, in addition to gold entities and time/table identification (ablation) | CheckEHR, GPT-3.5 (0613) component analysis · MIMIC-III, 25% test set | recall: 82.49% -> 92.83% | Helped |
Surprising findings
- Performance on MIMIC-OMOP was similar to or lower than on MIMIC-III despite OMOP's simpler dictionary-table structure, mainly due to complex entity names in the OMOP concept table.
- Mixtral scored zero in the zero-shot setting, so it is excluded from the zero-shot results.
Limitations
- MIMIC-III preprocessing for privacy may introduce inconsistencies absent in real hospital data.
- Recall remains in the 40-60% range, and NER is a major bottleneck, so the task is hard for current LLMs.
- Labels were not verified across all clinical notes in MIMIC-III; no repeated runs or error bars reported.
Experimental setup
Datasets: MIMIC-III MIMIC-OMOP C4-WSRS medical abbreviation dataset
Benchmarks: EHRCon (MIMIC-III) EHRCon (MIMIC-OMOP)
Metrics: Recall Precision Intersection
Baselines: GPT-3.5 (0613) Llama-3 70B Mixtral 8x7B Tulu2 70B
Builds on: StructGPT TabFact FEVEROUS
Closest papers by meaning
- Towards Error-Free EHRs: Reasoning-Intensive Consistency Verification Between Clinical Notes and Structured Tables in Electronic Health Records (NeurIPS 2026 · similarity 0.91)
- Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHR Data (ICLR 2025 · similarity 0.84)
- SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmark (NeurIPS 2024 · similarity 0.84)
- EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries (NeurIPS 2024 · similarity 0.83)
- Generating Multi-Table Time Series EHR from Latent Space with Minimal Preprocessing (NeurIPS 2025 · similarity 0.83)
- Multimodal Medical Code Tokenizer (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.