Research map / Large language models / Retrieval and knowledge
DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DiReCT, a physician-annotated benchmark of 511 MIMIC-IV clinical notes requiring multi-evidence entailment-tree diagnostic reasoning guided by a diagnostic knowledge graph, and a three-module LLM pipeline baseline evaluated on GPT-4 turbo, GPT-3.5 turbo, and open LLaMA3/Mistral/Mixtral/Zephyr models.
Problem
Existing medical LLM evaluations focus on short-text QA or NLI and do not assess whether models can find key observations in long clinical notes and reason step-by-step to a diagnosis in an interpretable way aligned with physicians.
Approach
A benchmark of clinical notes with physician-annotated multi-evidence entailment trees (observation, rationale, diagnosis) grounded in a guideline-derived diagnostic knowledge graph, plus a narrowing-down/perception/reasoning LLM pipeline that traverses the graph iteratively.
Key results
- GPT-4 turbo with the knowledge graph reaches 0.808 disease-category accuracy and 0.611 diagnosis accuracy, while LLama3 70B reaches 0.652 diagnosis accuracy with K.
- GPT-4 turbo achieves the best observation completeness (0.391) and explanation faithfulness (Exp_all 0.210 with G, 0.273 with K), but explanation scores remain low overall.
- Without any external knowledge, diagnosis accuracy collapses for all models (GPT-4 turbo 0.074), showing the benchmark's difficulty.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| provide the full knowledge graph K instead of procedural graph G | GPT-4 turbo diagnostic reasoning baseline · DiReCT | explanation completeness (all): 0.210 -> 0.273 | Helped |
| provide the full knowledge graph K instead of procedural graph G | GPT-4 turbo diagnostic reasoning baseline · DiReCT | observation completeness: 0.391 -> 0.371 | Hurt |
| provide the full knowledge graph K instead of procedural graph G | GPT-4 turbo diagnostic reasoning baseline · DiReCT | diagnosis accuracy: 0.610 -> 0.611 | Helped |
| provide the final-diagnosis set D* instead of no external knowledge | GPT-4 turbo diagnostic reasoning baseline · DiReCT | diagnosis accuracy: 0.074 -> 0.636 | Helped |
| add the missing observation to the note (ablation) | GPT-4 turbo with G · 73 amended cases | diagnosis accuracy: 0.328 -> 0.547 | Helped |
| add the missing observation to the note (ablation) | GPT-4 turbo with K · 73 amended cases | diagnosis accuracy: 0.260 -> 0.589 | Helped |
| use the full knowledge graph K instead of G on original notes | GPT-4 turbo · 73 amended cases, Original | diagnosis accuracy: 0.328 -> 0.260 | Hurt |
| use the full knowledge graph K instead of G on amended notes | GPT-4 turbo · 73 amended cases, Amended | diagnosis accuracy: 0.547 -> 0.589 | Helped |
Surprising findings
- Higher diagnostic accuracy does not imply better observation or explanation quality; LLama3 70B has high observation recall but low precision.
- Adding a single observation to 73 amended notes consistently improved all metrics, while supplying the full knowledge graph lowered diagnostic accuracy on original notes for some models.
Limitations
- Covers only a subset of disease categories (25) and a single primary discharge diagnosis, omitting inter-diagnostic relationships.
- Baseline pipeline may use suboptimal prompts and does not address hallucination; it performs poorly without the knowledge graph.
- Dataset is for model evaluation only and not for clinical use; data released via PhysioNet under MIMIC license.
Experimental setup
Datasets: MIMIC-IV
Benchmarks: DiReCT
Metrics: Acc_diag Acc_cat Obs_pre Obs_rec Obs_comp Exp_com Exp_all
Baselines: GPT-4 turbo GPT-3.5 turbo LLama3 70B LLama3 8B Mixtral 8x7B Mistral 7B Zephyr 7B
Builds on: entailment tree explanation chain-of-thought MIMIC-IV SOAP note format
Closest papers by meaning
- Large Language Models Are Clinical Reasoners: Reasoning-Aware Diagnosis Framework with Prompt-Generated Rationales (AAAI 2024 · similarity 0.89)
- ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room (NeurIPS 2026 · similarity 0.89)
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations (NeurIPS 2024 · similarity 0.88)
- CUPCase: Clinically Uncommon Patient Cases and Diagnoses Dataset (AAAI 2025 · similarity 0.87)
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.87)
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.