Research map / Vision-language and multimodal / Multimodal LLMs

ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ASCD, a training-free decoding method that positively steers attention in automatically mined text-centric heads and negatively steers the most-attended visual tokens, fused with contrastive decoding to reduce MLLM object hallucination.

Problem

MLLMs hallucinate by over-relying on spurious visual cues or language priors; existing contrastive decoding methods (VCD, ICD) reduce hallucination but their mechanism is opaque and they degrade general VQA performance.

Approach

Shows VCD and ICD gains coincide with cross-modal attention redistribution, then directly steers attention during decoding: boosts attention in model-specific text-centric heads (positive branch) and suppresses top-attended critical visual tokens (negative branch), combined via contrastive decoding with truncation.

Key results

• On LLaVA-1.5 7B greedy decoding, ASCD reduces CHAIRs from 53.2 to 35.6 (33.1% relative) and raises POPE accuracy from 85.37 to 86.53.
• On LLaVA-NeXT 7B beam decoding, ASCD cuts CHAIRs from 34.0 to 21.0 (38.2% relative), the largest reduction reported.
• On standard VQA, ASCD improves over original LLaVA-1.5 7B on ScienceQA (67.55 to 69.51) and TextVQA (57.82 to 57.91), while VCD degrades several benchmarks.

Surprising findings

• VCD and ICD degrade performance on several standard VQA benchmarks (e.g., VCD drops GQA from 61.28 to 59.38), while ASCD does not.
• Text-centric head identity is stable within a model across domains (COCO vs chest X-ray, intra-model JSD ~0.1) but differs strongly across architectures (JSD ~0.6).

Limitations

• Head selection is model-specific and must be profiled offline per backbone.
• Several reported gains are small on some metrics (e.g., POPE adversarial split with ASCD below Orig on some settings), and hyperparameters were tuned on CHAIR/POPE for LLaVA-1.5.
• Runtime overhead is measured only in eager attention path; SDPA and FlashAttention integration is not provided for ASCD.

Experimental setup

Datasets: COCO (500 images for head profiling) NIH ChestX-ray14 (500 images, domain robustness analysis)

Benchmarks: CHAIR POPE MMHal-Bench MM-VET ScienceQA GQA TextVQA MMMU

Metrics: CHAIRs CHAIRi POPE accuracy POPE F1 MMHal-Bench score VQA accuracy

Baselines: VCD ICD PAI Orig

Builds on: Visual Contrastive Decoding Instruction Contrastive Decoding PAI

Closest papers by meaning

• AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding (NeurIPS 2025 · similarity 0.92)
• Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Models (NeurIPS 2025 · similarity 0.92)
• Breaking the Illusion: When Positive Meets Negative in Multimodal Decoding (CVPR 2026 · similarity 0.92)
• ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models (AAAI 2025 · similarity 0.91)
• Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation (ICLR 2026 · similarity 0.91)
• The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs? (NeurIPS 2025 · similarity 0.91)