Research map / Vision-language and multimodal / Multimodal LLMs

Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Owl, a causally-grounded training-free decoding method that reweights visual and textual attention per token and layer guided by a new VTACR metric, and uses dual-path contrastive decoding to reduce object hallucination in LVLMs.

Problem

Object hallucination in large vision-language models, where generated objects are not supported by the image; existing attention-based decoding methods intervene on only one modality and ignore the visual-textual attention balance.

Approach

Models visual and textual attention as mediators in a structural causal model, defines VTACR (ratio of visual to textual attention contribution) to guide token- and layer-wise attention reweighting, and contrasts a vision-favored path with a text-favored path in a dual-path contrastive decoding step.

Key results

• On CHAIR with LLaVA-1.5, Owl reduces CHAIR_S to 26.2 and CHAIR_I to 8.1, versus 31.8 and 10.3 for PAI (the strongest prior method).
• On POPE adversarial setting, Owl reports 90.5 on LLaVA-1.5 and 83.4 on Shikra, the highest among compared methods in those columns.
• On VizWiz, Owl raises LLaVA-1.5 from 48.8 to 52.5 while VQAv2 drops slightly from 80.0 to 78.2.

Surprising findings

• Increasing textual attention worsens hallucination while increasing visual attention shortens outputs, a trade-off the method must balance.
• Owl slightly trails PAI on the POPE popular split for LLaVA-1.5 and MiniGPT-4.

Limitations

• Hyperparameters alpha, beta and lambda are tuned empirically per model.
• Evaluation uses only 500 sampled COCO images for generation benchmarks.
• The causal analysis relies on CHAIR as the outcome measure in the TCE definition.

Experimental setup

Datasets: MSCOCO val2014

Benchmarks: POPE CHAIR VQAv2 GQA VizWiz ScienceQA-IMG TextVQA

Metrics: CHAIR_S CHAIR_I caption length POPE accuracy F1 GPT-4V correctness and detailedness scores

Baselines: Beam Search Greedy Decoding Nucleus Sampling VCD PAI OPERA CausalMM

Builds on: visual contrastive decoding OPERA PAI CausalMM

Closest papers by meaning

• Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation (AAAI 2026 · similarity 0.95)
• Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination (CVPR 2026 · similarity 0.94)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.94)
• CausalLens: Sensitivity-Guided Multi-Head Causal Intervention for Hallucination Mitigation in Large Vision-Language Models (CVPR 2026 · similarity 0.94)
• One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination (CVPR 2026 · similarity 0.94)
• Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating (ICML 2026 · similarity 0.94)