Research map / Vision-language and multimodal / Multimodal LLMs
Imitating the Truth: Attention-aware Truth-Guided Enhancement for Hallucination Mitigation in Large Vision-Language Models
Hairui Ren, Zixuan Wang, Yibo Yang, He Zhao, Fan Tang, Dandan Guo, Yi Chang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AGE, an attention-aware truth-guided decoding-time enhancement that contrasts truthful and hallucinatory token dynamics to mitigate hallucinations in LVLMs.
Problem
LVLMs generate content inconsistent with visual evidence (hallucinations); existing mitigation relies on auxiliary modules or coarse decoding-time adjustments that overlook fine-grained differences between truthful and hallucinatory tokens.
Approach
Uses fine-grained attention dynamics to distinguish truthful (real) tokens from hallucinatory ones and guides decoding toward truthful tokens, without relying on auxiliary modules, as the abstract indicates; full method details are not available.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply AGE attention imitation at decoding | LLaVA-1.5-7B · COCO captioning | CHAIR_S: 20.80 -> 16.43 | Improved |
| apply AGE | MiniGPT-4-7B · COCO captioning | CHAIR_S: 30.87 -> 15.62 | Improved |
| apply AGE | MiniGPT-4-7B · POPE | accuracy: 56.77 -> 73.86 | Improved |
| apply AGE | LLaVA-1.5-13B · COCO captioning | CHAIR_I: 12.38 -> 12.55 | No clear effect |
| use directional image-attention vector instead of self-multiplicative amplification (ablation) | LLaVA-1.5-7B · COCO (128 tokens) | CHAIR_S: 43.1 -> 35.4 | Improved |
| add text attention intervention on top of image intervention (ablation) | LLaVA-1.5-7B · COCO (128 tokens) | CHAIR_S: 35.4 -> 31.8 | Improved |
| text attention intervention alone (ablation) | LLaVA-1.5-7B · COCO (128 tokens) | CHAIR_I: 14.2 -> 14.9 | Hurt |
| intervene on all middle-stage layers instead of one layer (ablation) | LLaVA-1.5-7B · COCO captioning | CHAIR_S: 16.43 -> 28.40 | Hurt |
| intervene on all late-stage layers instead of layers 30-31 (ablation) | LLaVA-1.5-7B · COCO captioning | CHAIR_S: 16.43 -> 19.20 | Hurt |
| compute delta at final layer rather than intervention layer (ablation) | LLaVA-1.5-7B · COCO captioning | CHAIR_S: 19.40 -> 16.43 | Improved |
| intervene image attention in early layers (1,2) (ablation) | LLaVA-1.5-7B · COCO captioning | BLEU: 15.93 -> 1.03 | Hurt |
| increase sample size M for delta from 5 to 200 (ablation) | AGE · COCO captioning | CHAIR_S | No clear effect |
Closest papers by meaning
- Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation (AAAI 2026 · similarity 0.94)
- Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models (ICLR 2026 · similarity 0.93)
- Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.93)
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
- AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM (CVPR 2026 · similarity 0.93)
- Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy (ICML 2026 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
