Research map / Vision-language and multimodal / Multimodal LLMs

Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CRG, a training-free inference-time method that splits each attention head into visual and text routes, estimates their decision-aligned effects with a one-forward/one-gradient approximation, and suppresses text routes in conflict heads to reduce LVLM hallucination.

Problem

LVLMs hallucinate fluent content unsupported by the image because language priors in the text pathway dominate the final token decision even when visual tokens receive attention; attention-share proxies (VAR) are correlational and coarse head rescaling harms grounded outputs.

Approach

Decompose each head's output exactly into visual-route and text-route components, measure do-style route effects on the decision score via a first-order gradient estimator, flag sign-conflicting heads, and apply rank-based text-route gating (mild for Conflict-A, strong for Conflict-B) while leaving visual routes intact.

Key results

• On POPE popular, CRG reaches 88.40 accuracy on LLaVA-1.5-7B vs 81.88 for Regular, and 87.63 on Qwen-VL-Chat vs 83.63.
• On CHAIR, LLaVA-1.5-7B C_S drops from 52.8 to 34.2 and C_I from 15.9 to 11.2 while recall stays near 77.8.
• On AMBER with LLaVA-1.5-7B, CHAIR falls from 8.3 to 4.6 and AMBER Score rises from 82.70 to 86.45.

Surprising findings

• About half of head-example pairs have negative visual-route effects (Pr=0.503), even among the top 10% VAR heads (0.515), suggesting high attention to visual tokens does not imply supportive visual evidence.

Limitations

• Route effect estimates are local to the current decoding step and may be noisy under ambiguous evidence or cross-layer interactions.
• Requires white-box access to activations and gradients, so it does not apply to closed-source APIs.
• Suppressing text routes may make outputs more conservative and reduce descriptive richness.

Experimental setup

Datasets: POPE CHAIR MME MMHal-Bench AMBER

Benchmarks: POPE CHAIR MME MMHal-Bench AMBER

Metrics: Accuracy F1-Score CHAIR C_S CHAIR C_I Recall MME score MMHal score AMBER Score

Baselines: Regular decoding VCD OPERA PAI VTI

Builds on: Visual Contrastive Decoding OPERA PAI VTI Causal Head Gating

Closest papers by meaning

• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.94)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.94)
• Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models (ICLR 2026 · similarity 0.93)
• Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.93)
• The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models Via Visual Information Steering (ICML 2025 · similarity 0.93)
• Envision, Attend, Then Respond: Counterfactual Hallucination Mitigation in Large Vision-Language Models (CVPR 2026 · similarity 0.93)