Research map / Vision-language and multimodal / Multimodal LLMs

Anatomical Region-Guided Contrastive Decoding: A Plug-and-Play Strategy for Mitigating Hallucinations in Medical VLMs

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ARCD, a training-free plug-and-play contrastive decoding method that uses anatomical region masks to re-weight tokens, attention, and logits of a medical VLM (Phi3.5V-Med) to reduce hallucinations.

Problem

Medical vision-language models hallucinate by relying on textual priors rather than visual evidence; existing training-based fixes need costly expert annotations and training-free contrastive decoding applies global, untargeted correction that is unreliable on complex clinical data.

Approach

Converts an anatomical segmentation mask into multi-scale token-level attention masks and applies three-tiered contrastive guidance between guided and unguided branches: token embedding suppression in the unguided branch, attention-probability reweighting by beta, and logit-level fusion with gamma.

Key results

• In zero-shot setting, ARCD raises Phi3.5V-Med overall accuracy on SLAKE from 50.00 to 55.11 (+5.11) and OBScan from 61.81 to 65.75 (+3.94), outperforming VCD, DoLa and OPERA.
• In the fine-tuned setting on MIMIC-Ext-VQA, ARCD improves overall accuracy from 69.29 to 77.95 (+8.66), the largest gain reported.
• Mask-based guidance gives the largest boost among prompt types in the ablation on SLAKE and OBScan (e.g., OBScan 65.74 with mask prompt vs 61.81 baseline).

Surprising findings

• Using only the class label as guidance hurt performance on OBScan (-8.67%), attributed to ambiguous clinical abbreviations.
• Simple center cropping gave a notable gain on MIMIC, and cropping methods harmed OBScan performance, suggesting lost textual context acts as a useful prior.

Limitations

• Requires expert or pretrained segmentation masks at inference, limiting plug-and-play applicability when masks are unavailable.
• Gains on MIMIC zero-shot are modest, and the fine-tuned model can generate plausible but unverified details on complex multi-disease reports.
• Evaluation relies on GPT-4o judging and a subset of MIMIC queries with bounding boxes; OBScan is not publicly released.

Experimental setup

Datasets: PubMedVision MIMIC-Ext-VQA SLAKE OBScan Chest-imagenome

Benchmarks: MIMIC-Ext-VQA SLAKE OBScan

Metrics: accuracy (closed-set) recall of ground-truth tokens (open-set) overall accuracy GPT-4o response category classification

Baselines: VCD DoLa OPERA LLaVA-1.6-7B Qwen-VL-7B GPT-4o LLaVA-Med-7B HuatuoV-34B

Builds on: visual contrastive decoding OPERA DoLa LoRA PubMedVision

Closest papers by meaning

• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.91)
• ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models (AAAI 2025 · similarity 0.91)
• SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.90)
• Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations (ICLR 2025 · similarity 0.90)
• Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.90)
• Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.90)