Research map / Vision-language and multimodal / Multimodal LLMs
Beyond Logits: Coherent Hallucination Mitigation via Attention Contrastive Decoding
Yujia Chen, Rui Sun, Huayu Mai, Wangkai Li, Zhangyu He, Bingzhou Wang, Aibing Li, Wenzhang SUN, Tianzhu Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes attention-level contrastive decoding for LVLMs as an alternative to logit-level contrastive decoding, aiming to reduce hallucination while preserving generation coherence.
Problem
LVLMs hallucinate factually inaccurate content; existing contrastive decoding mitigates this by contrasting amateur and expert branches at the logit level, which the authors argue compromises generation coherence and requires restrictive penalty constraints.
Approach
Moves the contrastive intervention from output logits to attention, contrasting expert and amateur attention so that hallucination mitigation does not damage coherence (details beyond the abstract not available).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ASS (adaptive subtraction) to attention contrastive decoding (ablation) | VCD+ACD on LLaVA-1.5 · CHAIR | CHAIRi: 15.6 -> 14.3 | Improved |
| add ASS to attention contrastive decoding (ablation) | VCD+ACD on LLaVA-1.5 · CHAIR | CHAIRs: 52.1 -> 51.0 | Improved |
| replace VCD logit penalty with uniform attention subtraction (ablation) | VCD on LLaVA-1.5 · CHAIR | CHAIRs: 51.8 vs 52.1 | No clear effect |
| add the penalty term to VCD (ablation) | VCD on LLaVA-1.5 · CHAIR | CHAIRs: 53.4 -> 51.8 | Improved |
| add ACD to VCD | VCD on LLaVA-1.5 · CHAIR | CHAIRi: 16.2 -> 14.3 | Improved |
| add ACD to ICD | ICD on LLaVA-1.5 · CHAIR | CHAIRi: 15.5 -> 13.9 | Improved |
| add ACD to SID | SID on LLaVA-1.5 · CHAIR | CHAIRi: 14.1 -> 13.3 | Improved |
| add ACD to VCD | VCD on Qwen-VL · CHAIR | CHAIRi: 13.7 -> 12.5 | Improved |
| add ACD to VCD | VCD on LLaVA-1.5 · LLaVA-Bench-Wild | total accuracy: 4.30 -> 6.10 | Improved |
| add ACD to VCD | VCD on Qwen-VL · LLaVA-Bench-Wild | total accuracy: 5.40 -> 7.11 | Improved |
| add ACD to VCD | VCD on LLaVA-1.5 · POPE Random | F1: 86.7 -> 87.2 | Improved |
| add ACD to VCD | VCD on LLaVA-1.5 · MME | Perception score: 1475.4 -> 1497.7 | Improved |
Surprising findings
- Logit-level contrastive decoding fundamentally compromises generation coherence, per the authors' investigation
Experimental setup
Baselines: contrastive decoding
Builds on: contrastive decoding
Closest papers by meaning
- SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.94)
- ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models (AAAI 2025 · similarity 0.94)
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.94)
- ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large Language Models (CVPR 2025 · similarity 0.93)
- The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs? (NeurIPS 2025 · similarity 0.93)
- Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
