Research map / Vision-language and multimodal / Multimodal LLMs

Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theory

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AFIP, a training-free decoding-time method that corrects attention distraction in MLLMs via cross-head attention enrichment and dynamic historical visual attention enhancement, with theoretical Rademacher-complexity bounds linking head heterogeneity to generalization.

Problem

Object hallucination in multimodal LLMs, whose visual perceptual causes (spatial inconsistency across attention heads and temporal fading of visual attention during decoding) are poorly understood, and existing mitigations need extra data or costly inference.

Approach

Draws an analogy between hallucination and human visual blur under divided attention; corrects attention by aggregating strong attention across top-k heads with a variance penalty, adds attention to visual tokens from the most influential historical text token, and gates the correction by the peak text-vs-visual attention gap.

Key results

• On CHAIR with LLaVA-1.5-7B (greedy decoding, 512 tokens), AFIP reaches C_S 16.8 and C_I 4.4 versus 24.6 and 6.4 for the strongest prior method HGAI.
• On POPE average accuracy, AFIP scores 87.11 on LLaVA-1.5-7B versus 85.72 for HGAI, and 86.90 on Shikra-7B versus 84.54 for HGAI.
• Averaged across five models on CHAIR, AFIP achieves C_S 17.5 and C_I 5.9, compared with 26.7 and 7.5 for HGAI.

Surprising findings

• Hallucinated tokens cluster in the lowest-30% visual attention ratio regions and appear mainly late in generation.
• Theorem 1's bound is minimized only when all heads have equal scale, so head heterogeneity increases the complexity bound.

Limitations

• Hyperparameters k and tau are sensitive and must be tuned per model.
• Analysis based on a small random COCO subset and mainly LLaVA-1.5-7B.
• Theoretical bounds rely on a simplified single-layer, CLS-token Transformer and loose assumptions.

Experimental setup

Datasets: COCO val (sampled subset for analysis)

Benchmarks: CHAIR POPE

Metrics: CHAIR C_S CHAIR C_I precision accuracy F1

Baselines: Greedy Beam search OPERA VCD MemVR HGAI

Builds on: OPERA VCD MemVR CHAIR POPE

Closest papers by meaning

• Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation (AAAI 2026 · similarity 0.92)
• Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.92)
• Beyond Attention Imbalance: Mitigating Hallucinations via Spectral Surgery (ICML 2026 · similarity 0.92)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.91)
• Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention (CVPR 2025 · similarity 0.91)
• ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large Language Models (CVPR 2025 · similarity 0.91)