Research map / Vision-language and multimodal / Multimodal LLMs

Enhancing Medical Large Vision-Language Models via Alignment Distillation

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MedAlign, a lightweight distillation framework that transfers patch-similarity structure and attention maps from a medical CLIP (UniMed-CLIP) into intermediate layers of Med-LVLMs via LoRA fine-tuning, improving report generation and medical VQA.

Problem

Med-LVLMs hallucinate because of insufficient visual representation learning (entangled, dispersed visual tokens) and misaligned visual attention toward clinically relevant regions.

Approach

Distill two signals from a domain-specific expert CLIP into one intermediate LLM layer: a spatial-aware loss matching pairwise cosine-similarity structure of patch tokens (after a learnable near-identity rotation and interpolation), and an attention-aware KL loss matching the visual attention distribution, all alongside LoRA fine-tuning without modifying the visual encoder.

Key results

• On IU-Xray report generation with HuatuoGPT-Vision-7B + LoRA, MedAlign reaches BLEU 10.73 and RaTEScore 60.49, best among the decoding and contrastive-decoding baselines (beam search BLEU 10.21).
• On medical VQA with LLaVA-Med-1.5 + LoRA, MedAlign improves SLAKE open-ended accuracy to 84.85 (vs. 83.50 best baseline PAI) and VQA-RAD open to 39.62.
• Removing either distillation loss reduces performance relative to the full method, and using UniMed-CLIP features without distillation degrades VQA-RAD open accuracy to 30.15 (vs. 36.95 LoRA-FT).

Surprising findings

• Directly plugging in UniMed-CLIP features without distillation degrades performance below the LoRA fine-tuning baseline.
• Many contrastive decoding methods (e.g., VCD, DAMRO, PAI) perform worse than or similar to plain greedy/beam decoding on Med-LVLMs without fine-tuning, with some reporting very low baseline report generation scores.

Limitations

• Assumes access to a high-quality, domain-relevant expert CLIP model; effectiveness depends on its quality.
• Alignment applied at only a single layer, so end-to-end optimization might yield further gains at higher cost.
• Interpolation across different token resolutions may introduce noise or inconsistencies.

Experimental setup

Datasets: MIMIC-CXR IU-Xray SLAKE VQA-RAD PathVQA OmniMedVQA

Benchmarks: IU-Xray MIMIC-CXR SLAKE VQA-RAD PathVQA OmniMedVQA

Metrics: BLEU ROUGE-L METEOR BERTScore CheXbert RadGraph RaTEScore accuracy recall

Baselines: Greedy decoding Beam search Nucleus sampling VCD DoLa OPERA AVISC M3ID DAMRO PAI A3Tune

Builds on: LoRA VCD OPERA A3Tune

Closest papers by meaning

• ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models (NeurIPS 2025 · similarity 0.90)
• Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering (CVPR 2025 · similarity 0.89)
• MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models (ICML 2026 · similarity 0.88)
• HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation (ICML 2025 · similarity 0.88)
• Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine (AAAI 2025 · similarity 0.88)
• OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.88)