Research map / Vision-language and multimodal / Multimodal LLMs

CausalLens: Sensitivity-Guided Multi-Head Causal Intervention for Hallucination Mitigation in Large Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CausalLens, a training-free, single-pass intervention that amplifies visually sensitive attention heads' visual pathway in mid-layers of LVLM decoders and applies a projection-aligned residual correction to reduce object hallucination.

Problem

LVLMs hallucinate objects and attributes that contradict the image because decoding is dominated by system prompt and textual priors; existing mitigation methods (contrastive decoding, multi-pass) add latency and memory.

Approach

Decompose each attention head's output into system, text, and visual pathway contributions, score heads by a visual sensitivity measure (normalized attention variance over image tokens), and in layers 10-20 shift each head toward its visual-minus-system contrast direction, followed by a residual correction projected through the output matrix W_O.

Key results

• On POPE (Popular split) with LLaVA-v1.5-7B, CausalLens reaches 86.5 accuracy and 86.8 F1 versus 82.3/82.1 for regular decoding and 84.8/84.9 for VAF.
• On CHAIR (max 64 tokens) with LLaVA-v1.5-7B, CausalLens achieves CHAIRs 18.7 and CHAIRi 6.2 versus 26.4 and 9.7 for regular decoding.
• Latency is 0.293 s per instance versus 0.281 s for regular decoding and 0.482 s for VCD, with peak GPU memory 16111 MB versus 15898 MB for regular decoding.

Surprising findings

• Ablating the top-k high-sensitivity heads drops POPE accuracy from 0.8793 to 0.5477 while ablating low-sensitivity heads leaves it essentially unchanged.

Limitations

• Intervention is restricted to a hand-chosen mid-layer range (layers 10-20) and a single coefficient lambda=0.15 tuned on POPE/CHAIR.
• Evaluation is limited to three LVLMs and mostly object-level hallucination benchmarks.
• The method's correctness rests on the assumption that the sensitivity score reflects causal visual reliability, supported only by a head-ablation analysis on POPE.

Experimental setup

Datasets: MS-COCO A-OKVQA GQA

Benchmarks: POPE MMHAL-Bench CHAIR MME LLaVA-Bench

Metrics: accuracy F1-score precision recall hallucination rate informativeness CHAIRs CHAIRi latency GPU memory

Baselines: Regular decoding VCD DeGF VAF

Builds on: visual contrastive decoding PAI attention amplification DeGF POPE

Closest papers by meaning

• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.94)
• Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.94)
• Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models (ICLR 2026 · similarity 0.93)
• Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation (ICML 2026 · similarity 0.93)
• Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality (ICLR 2025 · similarity 0.93)
• Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation (ICLR 2026 · similarity 0.93)