Research map / Vision-language and multimodal / Multimodal LLMs

Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Vision-Guided Attention (VGA), a training-free, FlashAttention-compatible method that builds token-level visual grounding from the model's own visual logits (Visual Semantic Confidence) and uses it to boost visual attention, reducing MLLM hallucinations.

Problem

MLLMs hallucinate because their visual attention localizes poorly and they underuse the semantic information already present in visual tokens; existing attention-based fixes rely on attention weights, which are incompatible with FlashAttention or need extra forward passes.

Approach

Derive a visual grounding from the logits that visual tokens produce over the vocabulary (Visual Semantic Confidence), then add this grounding as a bias to the last-token attention weights in intermediate layers, plus head balancing and, for captioning, a programmed grounding update that suppresses already-described regions.

Key results

• On POPE Random with LLaVA-7B, VGA reaches 89.28 accuracy vs 87.12 for vanilla and 87.27 for PAI_CD.
• On CHAIR with LLaVA-7B, VGA lowers CHAIRs from 52.2 (vanilla) to 29.8 with F1 76.3.
• On AMBER with LLaVA-7B, VGA achieves the best AMBER score of 38.00 vs 34.35 vanilla; latency overhead is 4.36% TTFT and 6.01% TPOT.

Surprising findings

• Grounding from negative/absent-object queries still improves rejection of absent objects, and spurious low-Dice grounding (DICE<0.01) still gave 88.5% accuracy vs 87.1% for uniform guidance, which the authors attribute to information flow between visual tokens.

Limitations

• Failure cases occur when the model fails to extract visual features, so VSC-derived grounding is poor.
• Hyperparameters (beta, lambda, starting and ending layers) are tuned per model and benchmark, with some tuning on validation sets.
• Evaluated on only three MLLMs, all of them LLaVA or Qwen2.5-VL variants.

Experimental setup

Datasets: MSCOCO A-OKVQA GQA

Benchmarks: POPE CHAIR AMBER MME

Metrics: accuracy F1 CHAIRs CHAIRi AMBER score Cover Hal Cog TTFT TPOT

Baselines: PAI PAI_CD VAF TARAC Vanilla

Builds on: PAI TARAC VAF

Closest papers by meaning

• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.92)
• Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation (AAAI 2026 · similarity 0.92)
• One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination (CVPR 2026 · similarity 0.92)
• Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.92)
• Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy (ICML 2026 · similarity 0.92)
• Seeing Is Believing: Rich-Context Hallucination Detection for MLLMs via Backward Visual Grounding (AAAI 2026 · similarity 0.91)