Research map / Vision-language and multimodal / Multimodal LLMs

ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ExGra-Med, a multi-graph alignment objective over image, instruction-response, and GPT-4-extended caption embeddings, trained end-to-end on LLaMA-7B via black-box (IMLE) gradient estimation, to improve data efficiency of medical vision-language models.

Problem

Medical multimodal LLMs such as LLaVA-Med rely on autoregressive training and large instruction-following data; with limited data their vision-language alignment is weak and downstream performance drops sharply.

Approach

Replace projector-only alignment with a second-order multi-graph alignment that forms triplet correspondences among image, original answer, and GPT-4-extended answer graphs via a barycenter graph, and backpropagates through the non-differentiable matching solver using IMLE-style black-box gradient estimation so the full LLM can be trained.

Key results

• With 10% of pre-training data, ExGra-Med reaches 72.52% average on VQA-RAD versus 52.39% for LLaVA-Med (10%) and 72.64% for LLaVA-Med trained on 100%.
• With full data, ExGra-Med (DCI) achieves 75.1% overall and 64.82% PathVQA average, versus 73.37% overall for LLaVA-Med.
• Ablation: removing the extended context drops VQA-RAD average from 74.37% to 72.12% (40% setting).

Measured effects reported in the paper

Surprising findings

• LLaVA-Med trained on 10% of pre-training data drops from 72.64% to 52.39% on VQA-RAD, indicating autoregressive training is highly data-hungry.
• Using extended captions alone in LLaVA-Med gives only partial gains (69.41% at 10%), far below ExGra-Med's 72.52%.

Limitations

• Experiments limited to the LLaVA architecture; not validated on other architectures such as Flamingo.
• Extended captions generated by GPT-4 may contain hallucinations; user study shows mostly good but not perfect quality.
• Some baseline numbers are taken from literature rather than re-run under identical settings.

Experimental setup

Datasets: PMC-15M (via LLaVA-Med) VQA-RAD SLAKE PathVQA

Benchmarks: VQA-RAD SLAKE PathVQA

Metrics: recall (open-ended) accuracy (closed-ended) average accuracy overall score

Baselines: LLaVA-Med InfoNCE PLOT SigLIP VLAP GeoCLAP PAC-S ImageBind BiomedGPT Med-Dr Med-MoE RadFM

Builds on: LLaVA-Med LLaVA IMLE InfoNCE CLIP

Closest papers by meaning

• Enhancing Medical Large Vision-Language Models via Alignment Distillation (AAAI 2026 · similarity 0.90)
• LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding (CVPR 2026 · similarity 0.89)
• LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation (ICLR 2024 · similarity 0.88)
• Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.88)
• MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization (ICML 2025 · similarity 0.88)
• GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and a Comprehensive Multimodal Dataset Towards General Medical AI (AAAI 2026 · similarity 0.88)