Research map / Large language models / Language, safety and interpretability
Large Language Models Are Clinical Reasoners: Reasoning-Aware Diagnosis Framework with Prompt-Generated Rationales
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a reasoning-aware Alzheimer's diagnosis framework where GPT-4 generates clinical chain-of-thought rationales from patient data, which are then distilled into OPT/LLaMA2 unimodal and 3D-ResNet+LM multimodal students, and introduces five clinician-defined criteria for rationale quality.
Problem
Disease diagnosis models are typically image/text classifiers that lack explicit clinical reasoning and explanations, and expert-written rationales are expensive to annotate, so clinical reasoning for diagnosis is under-explored with LLMs.
Approach
Prompt GPT-4 with a few clinician-selected rationale demonstrations to generate Clinical CoT rationales conditioned on ground-truth diagnosis, then fine-tune smaller unimodal and multimodal language models to generate rationale plus diagnosis, and evaluate rationales with new clinical criteria.
Key results
- Clinical CoT with GPT-4 reaches 68.4% in-domain ADNI accuracy versus 59.6% zero-shot and 62.6% 5-shot.
- Distilled LLaMA2 7B student reaches 71.5% ADNI accuracy, above the GPT-4 teacher's 68.4%, and OPT 1.3B reaches 70.0% on out-of-domain AIBL versus 62.6% for the teacher.
- Multimodal students (3D ResNet + OPT 6.7B) reach up to 71.0% ADNI accuracy, and with 25% of training data outperform baselines trained on 100%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use two-shot Clinical CoT prompting instead of standard few-shot prompting | ChatGPT · ADNI | accuracy: 57.3 -> 67.3 | Helped |
| Use two-shot Clinical CoT prompting instead of standard few-shot prompting | ChatGPT · AIBL | accuracy: 57.2 -> 62.4 | Helped |
| Use two-shot Clinical CoT prompting instead of standard few-shot prompting | GPT-4 · ADNI | accuracy: 62.6 -> 68.4 | Helped |
| Use two-shot Clinical CoT prompting instead of standard few-shot prompting | GPT-4 · AIBL | accuracy: 59.8 -> 62.6 | Helped |
| Distill using clinical rationales rather than training without rationale augmentation (ablation) | Student diagnosis models · ADNI | accuracy | Helped |
| Train the multimodal student with clinical rationales using only 25% of training data (ablation) | Multimodal student · ADNI | accuracy | Helped |
| Use a multimodal student instead of GPT-4 | Multimodal student · ADNI | accuracy: 68.4 vs 68.6; 68.4 vs 70.8; 68.4 vs 69.0; 68.4 vs 68.9; 68.4 vs 71.0; 68.4 vs 68.5 | Mixed |
| Use a unimodal student instead of GPT-4 | Unimodal student · ADNI | accuracy: 68.4 vs 66.5; 68.4 vs 68.4; 68.4 vs 71.5 | Mixed |
| Use a unimodal student instead of GPT-4 | Unimodal student · AIBL | accuracy: 62.6 vs 70.0; 62.6 vs 66.1; 62.6 vs 69.4 | Helped |
| Use a multimodal student instead of a vision-only baseline | Multimodal student · ADNI | accuracy: 49.8 vs 68.6; 49.8 vs 70.8; 49.8 vs 69.0; 51.9 vs 68.9; 51.9 vs 71.0; 51.9 vs 68.5 | Helped |
Surprising findings
- Rationales generated without access to the ground-truth diagnosis were rated higher than those conditioned on it for misdiagnosed cases.
- Text-only unimodal students outperform vision-only 3D ResNet baselines in total accuracy despite AD diagnosis being image-intensive.
Limitations
- Only two rationale demonstrations fit in the prompt due to length, potentially limiting rationale quality.
- No filtering of ineffective rationales; 25% of misdiagnosis rationales contained medically incorrect knowledge.
- Not deployed in real clinical settings; uses a single testbed (Alzheimer's disease).
Experimental setup
Datasets: ADNI AIBL
Benchmarks: ADNI AIBL
Metrics: accuracy precision recall
Baselines: ChatGPT few-shot prompting GPT-4 few-shot prompting 3D ResNet-50 3D ResNet-152
Builds on: chain-of-thought prompting multimodal chain-of-thought knowledge distillation
Closest papers by meaning
- DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Models (NeurIPS 2024 · similarity 0.89)
- Memorize and Rank: Elevating Large Language Models for Clinical Diagnosis Prediction (AAAI 2025 · similarity 0.89)
- Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.88)
- ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room (NeurIPS 2026 · similarity 0.87)
- Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search (NeurIPS 2025 · similarity 0.87)
- MedS³: Towards Medical Slow Thinking with Self-Evolved Soft Dual-sided Process Supervision (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.