Research map / Vision-language and multimodal / Multimodal LLMs

Dual-Pathway Circuits of Object Hallucination in Vision-Language Models

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Applies activation patching and a new Conditional Pathway Analysis to five VLMs to find grounding and hallucination pathways behind POPE-adversarial object hallucination, then suppresses hallucination-pathway components to cut hallucination by up to 76%.

Problem

Object hallucination in vision-language models is typically treated as a black-box failure; the internal circuits producing it, and whether they are shared across VLM architectures, are unclear.

Approach

Activation patching with corrupted visual tokens splits components into a visual grounding pathway and a hallucination pathway; Conditional Pathway Analysis measures within-pathway redundancy and reveals a polarity flip of grounding-component IEs between correct and hallucinating samples; per-component scaling of hallucination-pathway outputs causally reduces hallucination.

Key results

• Scaling hallucination-pathway components reduces POPE-adversarial hallucination by up to 76% (Llama-3.2-Vision) at ≤2 pp accuracy cost.
• Grounding-pathway components show a consistent IE polarity flip (subset-mean IE positive on correct samples, negative on hallucinating ones) in all five models, with Cohen's d from -1.60 to -2.45.
• Random-component suppression slightly increases hallucination (mean +3.0 pp), while ITI gives a mean 4.1 pp reduction versus 7.4 pp for component scaling.

Surprising findings

• Grounding-pathway ablation at s=0 collapses accuracy to chance in every model, while hallucination-pathway suppression reduces hallucination with little accuracy cost.
• Mean-difference projection yields essentially no reduction (0.8 pp mean), and the probe-based ITI direction points toward hallucination at positive alpha.

Limitations

• Per-component scaling is a causal test rather than a deployable mitigation method, as the authors note.
• Attribute hallucination on AMBER is largely unaffected, so the circuit transfers only to existence and relational types.
• Analysis covers only five VLMs and object-existence questions on POPE-adversarial with a 400-sample test set.

Experimental setup

Datasets: POPE-adversarial POPE-popular POPE-random AMBER

Benchmarks: POPE-adversarial POPE-popular POPE-random AMBER

Metrics: hallucination rate accuracy indirect effect Cohen's d logit lens delta

Baselines: Inference-Time Intervention (ITI) mean-difference projection random component control grounding-pathway ablation

Builds on: activation patching ITI logit lens

Closest papers by meaning

• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.94)
• Multi-Object Hallucination in Vision Language Models (NeurIPS 2024 · similarity 0.92)
• Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models (ICLR 2026 · similarity 0.92)
• Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating (ICML 2026 · similarity 0.92)
• Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.92)
• SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.91)