Research map / Vision-language and multimodal / Multimodal LLMs

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ILVAD, a training-free, plug-and-play inference-time method that builds a visual-evidence saliency map from inter-layer attention discrepancy and uses it to enhance visual and evidence-grounded text attention, reducing hallucinations in LVLMs.

Problem

LVLMs hallucinate when they insufficiently attend to correct visual evidence and progressively forget it during long generation; existing attention-based and decoding fixes do not reliably keep attention on query-relevant evidence.

Approach

Identify visual evidence tokens as those repeatedly newly activated (salient) across layers, which filters out attention sinks; use the resulting saliency map to amplify attention to evidence tokens during decoding and to up-weight generated text tokens strongly attending to evidence.

Key results

• On LLaVA-1.5-7B, ILVAD reduces CHAIR_S from 48.6 (greedy) to 32.6 and CHAIR_I from 13.52 to 9.42.
• ILVAD improves POPE accuracy on LLaVA-1.5-7B from 84.50 to 85.76 and reduces MMHal-Bench hallucination count from 67.0 to 61.5.
• On Qwen3-VL-8B, CHAIR_S drops from 51.6 to 45.2 and MMHal hallucination count from 35.4 to 32.6, with near-zero inference overhead claimed.

Surprising findings

• Enhancing only text tokens without visual evidence enhancement does not improve visual understanding (w/o Visual ablation shows no gain on CHAIR/POPE).
• A smaller first-token window T (1 or 5) gives better CHAIR_S/POPE but abnormally short captions, showing a trade-off with caption length.

Limitations

• Hyperparameters (tau, alpha, beta, T) are model- and benchmark-dependent and require tuning; beta must be chosen per benchmark.
• Gains on POPE F1 are not uniformly best; on some benchmarks (e.g., POPE F1 for LLaVA-1.5) AGLA performs better.
• Evaluation relies on GPT-4o as judge for several benchmarks, and CHAIR uses only 500 images.

Experimental setup

Datasets: MSCOCO val (500 images for CHAIR) IIW-400

Benchmarks: CHAIR POPE MMHal-Bench MME LLaVA-Bench CapMAS

Metrics: CHAIR_S CHAIR_I caption length POPE accuracy POPE F1 MMHal score hallucination rate MME total LLaVA-Bench accuracy/detailedness/naturalness ALOHa

Baselines: Greedy Beam search VCD CODE AGLA ONLY VAF VAR SPARC VHR

Builds on: visual attention sink contrastive decoding attention steering

Closest papers by meaning

• Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.94)
• Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination (CVPR 2026 · similarity 0.94)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.94)
• Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation (AAAI 2026 · similarity 0.94)
• Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling (NeurIPS 2025 · similarity 0.93)
• Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.93)