Research map / Vision-language and multimodal / Visual question answering
Visual and Domain Knowledge for Professional-level Graph-of-Thought Medical Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HIE-Reasoning, a expert-annotated neonatal HIE MRI benchmark with six clinical reasoning tasks, and CGoT, a clinical graph-of-thought prompting method that feeds visual and textual clinical knowledge to LVLMs.
Problem
Existing medical VQA datasets test basic visual perception, not the complex clinical reasoning (lesion grading, anatomy, injury score, 2-year neurocognitive outcome prediction) used by clinicians for HIE diagnosis and prognosis.
Approach
Decompose the complex outcome prediction into a graph of simpler clinical sub-tasks (lesion grading, lesion anatomy, rare-location lesions, MRI injury score), injecting visual knowledge (ADC, ZADC lesion probability, patient-specific brain ROI maps) and meta/individual textual clinical knowledge, with each step's answer feeding the next.
Key results
- CGoT-Gemini-1.5-Flash reaches 71.73% accuracy on two-year neurocognitive outcome prediction versus a 56.60% inter-class accuracy style baseline range; direct LVLM baselines score near chance (e.g., GPT4o-Series 52.16% inter-class accuracy).
- CGoT improves lesion grading accuracy from 33.83% (GPT4o-Series) to 56.25% and from 24.81% (Gemini-1.5-Flash) to 62.41% with CGoT.
- Ablation shows removing the MRI injury score from the reasoning chain drops outcome accuracy from 71.73% to 50.94%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CGoT clinical reasoning and clinical knowledge | Gemini-1.5-Flash · HIE-Reasoning | Neurocognitive Outcome inter-class accuracy: 56.60% -> 71.73% | Helped |
| Add clinical knowledge (ablation) | CGoT · HIE-Reasoning | Neurocognitive Outcome inter-class accuracy: 51.89% -> 71.73% | Helped |
| Add graph of thoughts (ablation) | CGoT · HIE-Reasoning | Neurocognitive Outcome inter-class accuracy: 52.43% -> 71.73% | Helped |
| Add clinical knowledge and graph of thoughts (ablation) | CGoT · HIE-Reasoning | Neurocognitive Outcome inter-class accuracy: 54.44% -> 71.73% | Helped |
| Add MRI Injury Score reasoning task (ablation) | CGoT · HIE-Reasoning | Neurocognitive Outcome inter-class accuracy: 50.94% -> 71.73% | Helped |
| Add Lesion Anatomy reasoning task (ablation) | CGoT · HIE-Reasoning | Neurocognitive Outcome inter-class accuracy: 70.12% -> 71.73% | Helped |
| Add Lesion Grading reasoning task (ablation) | CGoT · HIE-Reasoning | Neurocognitive Outcome inter-class accuracy: 61.11% -> 71.73% | Helped |
| Perturb predicted MRI Injury Score by ±1 level in 10% of test cases (ablation) | CGoT · HIE-Reasoning | Neurocognitive Outcome inter-class accuracy: 71.73 -> 67.83 | Hurt |
| Perturb predicted MRI Injury Score by ±1 level in 20% of test cases (ablation) | CGoT · HIE-Reasoning | Neurocognitive Outcome inter-class accuracy: 71.73 -> 66.22 | Hurt |
| Perturb predicted MRI Injury Score by ±1 level in 30% of test cases (ablation) | CGoT · HIE-Reasoning | Neurocognitive Outcome inter-class accuracy: 71.73 -> 62.72 | Hurt |
Surprising findings
- Direct LVLM outputs on the outcome task are comparable to or worse than random guessing, and GPT-4o models refuse some questions due to conservative alignment.
- Removing clinical knowledge and the graph of thoughts both drop outcome accuracy to roughly 52-54%, close to the no-knowledge baseline.
Limitations
- Small dataset of 133 individuals from a single institution (MGH), limiting generalizability.
- Outcome prediction task has only 84 QA pairs; ground truth labels are imbalanced.
- Lesion ROIs and lesion masks are generated by thresholding ZADC maps and automated ROI registration, which may be inaccurate.
Experimental setup
Datasets: HIE-Reasoning
Benchmarks: HIE-Reasoning
Metrics: accuracy MAE F1 score inter-class accuracy ROUGE-L
Baselines: GPT-4o GPT-4o-mini Gemini-1.5-Flash MiniGPT4-Med LLaVA-Med Med-Flamingo
Builds on: graph-of-thought chain-of-thought LLaVA-Med Med-Flamingo
Closest papers by meaning
- Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning (CVPR 2026 · similarity 0.91)
- OpenMedReason: Scientific Reasoning Supervision for Medical Vision–Language Models (NeurIPS 2026 · similarity 0.90)
- Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks (ICLR 2026 · similarity 0.90)
- Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search (NeurIPS 2025 · similarity 0.90)
- Falcon Medical Visual Question Answering (AAAI 2025 · similarity 0.90)
- MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text (AAAI 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.