Research map / Vision-language and multimodal / Human and embodied modeling

LLaVA Needs More Knowledge: Retrieval Augmented Natural Language Generation with Knowledge Graph for Explaining Thoracic Pathologies

AAAI 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Medical vision-language models · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes KG-RAG, a privacy-preserving knowledge-graph retrieval module (RadGraph triplets from MIMIC-CXR, retrieved via MedCLIP cross-modal search) plugged into three VLM frameworks (KG-LLaVA, Med-XPT, Bio-LLaVA) to generate natural language explanations for chest X-ray pathologies on MIMIC-NLE.

Problem

Generating natural language explanations for thoracic pathologies in chest X-rays, where general models lack medical domain knowledge and retrieval-based augmentation risks patient data leakage.

Approach

Replace patient-level retrieval datastores with a knowledge graph of de-identified RadGraph 'suggestive_of' triplets, embedded with MedCLIP's text encoder and retrieved by cross-modal image-to-triplet similarity, so no image or patient-specific data is stored; the module is plug-and-play across VLM architectures.

Key results

• KG-LLaVA reaches AUC 83.0 and CIDEr 62.2 on the MIMIC-NLE test set, versus 66.4 AUC for RATCHET (best prior AUC).
• KG-LLaVA achieves BLEU-4 7.2 and ROUGE-L 25.0, the highest among compared methods on those metrics.
• Med-XPT with KG retrieval achieves the best CIDEr of 62.7, slightly above KG-LLaVA's 62.2, while KG-LLaVA leads in METEOR (15.1 vs 14.3 for Bio-LLaVA).

Measured effects reported in the paper

Surprising findings

• Cross-modal KG retrieval (image-to-triplet) outperforms uni-modal image-to-image retrieval (CIDEr 62.2 vs 49.9), and the paper argues it also avoids storing patient images.
• Med-XPT with KG retrieval scores CIDEr 62.7, higher than KG-LLaVA's 62.2, despite Med-XPT using a smaller GPT-2 backbone.

Limitations

• Depends on a KG extraction model such as RadGraph; new modalities require adapting or training alternative graph-processing models.
• Failure cases show generic or repeated triplets cause localization errors and default diagnoses.
• Needs further clinical validation before real-world deployment.

Experimental setup

Datasets: MIMIC-NLE MIMIC-CXR

Benchmarks: MIMIC-NLE

Metrics: AUC BLEU-4 METEOR ROUGE-L CIDEr

Baselines: RATCHET TieNet DPT LoRA AE Prompt AE Prefix AE LLaMA-Adapt AE

Builds on: LLaVA MedCLIP RadGraph DPT RAG

Closest papers by meaning

• Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrieval (ICLR 2025 · similarity 0.87)
• Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation (NeurIPS 2025 · similarity 0.87)
• LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding (CVPR 2026 · similarity 0.86)
• Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Models (CVPR 2024 · similarity 0.86)
• OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.86)
• HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation (ICML 2025 · similarity 0.86)