Research map / Vision-language and multimodal / Multimodal LLMs

AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AdaIAT, a training-free attention-intervention method for LVLMs that adaptively amplifies attention to generated text and image tokens per head to reduce object hallucinations without inducing repetitive captions.

Problem

Hallucination in large vision-language models; naively increasing attention to image tokens reduces hallucinations but causes repetitive descriptions because attention to generated text is insufficient.

Approach

Observes that true objects receive higher attention over both image and text-prompt tokens than hallucinated ones, and that attention-head behavior varies widely; AdaIAT adaptively increases attention to image and generated-text tokens per attention head, rather than uniformly amplifying image attention as in PAI/IAT/HGAI.

Key results

• On LLaVA-1.5-7B, AdaIAT reaches F1 79.4 with C_S 31.4 and C_I 8.3, versus Greedy C_S 49.0 and F1 77.9, at about 114 ms/token (Greedy 109.8).
• On OpenCHAIR, AdaIAT achieves the lowest C_O on LLaVA-1.5-13B (0.235 vs Greedy 0.283) and Janus-Pro-7B (0.298 vs 0.327), but PAI is better on Qwen2.5-VL (0.303 vs 0.315).
• Contrastive methods VCD and AGLA cost 3-4x Greedy latency; LURE costs about 9x Greedy latency at 916 ms/token.

Surprising findings

• Uniformly increasing attention to image tokens reduces hallucination but causes repetitive descriptions; for some models large amplification factors make generation fail entirely.
• Attention-head disparity between true and hallucinated objects varies from 2x to 15x across heads and models.

Limitations

• Hyperparameters (alpha, beta) must be tuned per model and per method, and results are reported at chosen settings.
• On Qwen2.5-VL, AdaIAT slightly underperforms PAI on OpenCHAIR C_O.

Experimental setup

Datasets: COCO2014 OpenCHAIR

Benchmarks: CHAIR OpenCHAIR

Metrics: CHAIR C_S CHAIR C_I F1 D1 distinct-1 OpenCHAIR C_O latency ms/token

Baselines: Greedy decoding VCD AGLA OPERA LURE PAI HGAI IAT

Builds on: PAI IAT HGAI OpenCHAIR

Closest papers by meaning

• Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation (AAAI 2026 · similarity 0.93)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.93)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
• Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models (ICLR 2025 · similarity 0.93)
• Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy (ICML 2026 · similarity 0.93)
• Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.93)