Research map / Vision-language and multimodal / Multimodal LLMs

Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VisFlow, a training-free inference-time method that modulates decoder attention at token level (boosting salient visual tokens, RoPE-bias correction) and head level (suppressing system and text heads) to reduce visual hallucination in LVLMs.

Problem

Large vision-language models often hallucinate objects not present in the image, due to insufficient attention to visual tokens and over-reliance on language priors, while existing mitigations need retraining or add costly multi-pass decoding.

Approach

Identifies visual sink and salient tokens via intra-visual reception scores and classifies attention heads into visual, system, and text types; then rescales attention (amplify salient visual tokens, attenuate sinks, downweight system/text heads) and renormalizes, all in a single decoding pass.

Key results

• On POPE with LLaVA-1.5, VisFlow reaches 89.55 F1 (Random), 87.09 (Popular), 84.35 (Adversarial), above the best listed baseline (PAI* 85.94 Random).
• On CHAIR with LLaVA-1.5, VisFlow lowers CHAIR_i from 20.0 (greedy) to 15.0 and CHAIR_s from 6.8 to 3.8 while raising recall from 59.1 to 63.1.
• Masking visual-sensitive heads raises LLaVA-1.5 CHAIR_s from 20.0 to 46.0, while masking system heads has minimal effect (20.0).

Surprising findings

• Masking system-dominant heads has negligible effect on hallucination metrics, suggesting redundancy, while masking shallow text heads reduces CHAIR_s from 20.0 to 15.0.
• Removing HAI for system heads gives the lowest CHAIR_s (12.0) but reduces recall, indicating a trade-off.

Limitations

• Hyperparameters (k, delta, alpha, lambda) are set per model and per benchmark, with tuning on evaluation settings.
• TAI is omitted for models with compressed visual tokens (MiniGPT-4, mPLUG-Owl2) due to limited effectiveness.
• Evaluated on only three LVLMs and a small GPT-4.1-judged set of 60 questions.

Experimental setup

Datasets: MSCOCO POPE CHAIR LLaVA-Bench (in-the-wild)

Benchmarks: POPE CHAIR LLaVA-Bench (in-the-wild)

Metrics: F1 CHAIR_i CHAIR_s Recall GPT-4.1 accuracy/detailedness score TPS

Baselines: Greedy Beam Search OPERA VCD DoLa PAI VAR

Builds on: VAR PAI VCD OPERA DoLa

Closest papers by meaning

• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.95)
• Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy (ICML 2026 · similarity 0.94)
• Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.94)
• One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination (CVPR 2026 · similarity 0.93)
• AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM (CVPR 2026 · similarity 0.93)
• Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.93)