Research map / Vision-language and multimodal / Multimodal LLMs

Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Fox, a training-free inference-time method that localizes "risky" attention heads via a visual-attention-entropy times system-prompt-attention score, intervenes by saturating their pre-softmax logits, and fuses observational and interventional logits with a conflict-gated JSD rule to reduce object hallucination in LVLMs.

Problem

Object hallucination in large vision-language models during decoding; prior inference-time methods assume hallucination stems from a global deficit in visual attention intensity, which the authors argue is incomplete.

Approach

Replace the attention-intensity assumption with a structural view: hallucination arises when specific heads at decision-critical steps decouple from visual evidence and attend to system-prompt tokens. Fox scores heads by system-attention mass times visual-attention entropy, suppresses the top-scoring heads via a large negative logit bias (a do-operator), and gates the combination of the original a

Key results

• On CHAIR, Fox reduces instance-level CHAIR_I to 12.90 on LLaVA-1.5 and 11.98 on InstructBLIP, a 29.1% relative reduction over SID on InstructBLIP per the paper.
• On POPE adversarial, Fox reaches 81.93% accuracy on LLaVA-1.5, the best among compared methods in that setting.
• On MME, Fox raises LLaVA-1.5's total score to 613.33 versus 600.00 for SID.

Surprising findings

• Global visual attention mass shows no statistically significant difference between faithful and hallucinated outputs, challenging the attention-intensity assumption.
• Intervening at late layers yields diminishing returns, with early-to-mid layers (2-13) being the more effective intervention window.

Limitations

• Evaluated only on object-centric hallucination in three LVLMs; compositional, relational and video settings are left to future work.
• Requires dual forward passes per decoding step, adding inference cost relative to plain decoding.
• Hyperparameters (k, tau_JS, beta) are tuned per model via grid search on a validation set.

Experimental setup

Datasets: COCO POPE CHAIR MME

Benchmarks: POPE CHAIR MME

Metrics: accuracy F1 CHAIR_S CHAIR_I MME score GPT-4V correctness and detailedness scores

Baselines: Sampling VCD ICD OPERA CausalMM SID

Builds on: VCD OPERA PAI CausalMM

Closest papers by meaning

• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.92)
• Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination (CVPR 2026 · similarity 0.92)
• Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating (ICML 2026 · similarity 0.92)
• Mitigating Object Hallucination via Concentric Causal Attention (NeurIPS 2024 · similarity 0.92)
• The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models Via Visual Information Steering (ICML 2025 · similarity 0.91)
• CausalLens: Sensitivity-Guided Multi-Head Causal Intervention for Hallucination Mitigation in Large Vision-Language Models (CVPR 2026 · similarity 0.91)