Research map / Vision-language and multimodal / Multimodal LLMs

Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FCCT, a Gaussian-noise causal tracing analysis of LVLM object perception across visual/textual token types, MHSA, FFN and hidden states, and IRI, a training-free inference-time method that injects recovery-rate-scaled mid-layer MHSA/MLP outputs to reduce object hallucination.

Problem

Mechanistic interpretability of how LVLMs process visual object information across token types, components and layers is underexplored, and existing hallucination mitigation methods lack principled layer/component selection.

Approach

Uses activation patching with Gaussian-noised images and a Recovery Rate metric to localize causal effects of MHSA, MLP and hidden states over seven token categories and all layers, then injects the last-token MHSA and MLP outputs from high-RR intermediate layers into later layers, norm-preserved, at inference time.

Key results

• On POPE, IRI improves average accuracy by +4.89% and F1 by +5.20% over regular decoding across LVLMs and settings.
• On LLaVA-1.5-7b CHAIR, IRI reduces C_S from 52.8 to 34.6 and C_I from 15.9 to 11.5.
• On Qwen2-VL-7B, IRI raises POPE accuracy from 88.49 to 89.04 and lowers CHAIR C_S from 24.8 to 14.2.

Surprising findings

• Intervening directly on hidden states with the IRI scheme degrades performance relative to intervening on MHSA and MLP outputs separately.

Limitations

• Causal tracing uses only 500 COCO images and binary existence questions, limiting generality.
• Injection scaling coefficients are tuned per model, requiring hyperparameter search.

Experimental setup

Datasets: COCO

Benchmarks: POPE MME CHAIR MMHal-Bench MHumanEval

Metrics: accuracy F1 MME score CHAIR C_S CHAIR C_I MMHal score VH rate hallucination rate TTFT TPOT

Baselines: VCD OPERA PAI VTI

Builds on: ROME causal tracing attention knockout NOTICE VCD OPERA PAI VTI

Closest papers by meaning

• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.92)
• CausalLens: Sensitivity-Guided Multi-Head Causal Intervention for Hallucination Mitigation in Large Vision-Language Models (CVPR 2026 · similarity 0.92)
• The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models Via Visual Information Steering (ICML 2025 · similarity 0.92)
• Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.92)
• Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination (CVPR 2026 · similarity 0.91)
• First Logit Boosting: Visual Grounding Method to Mitigate Object Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.91)