Research map / Vision-language and multimodal / Multimodal LLMs

Causal-HalBench: Uncovering LVLMs Object Hallucinations Through Causal Intervention

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces a causal framework (SCM + visual content intervention) for LVLM object hallucination and builds Causal-HalBench, a counterfactual image benchmark with ACE/DCS-style metrics (CAC, AAC, CHR) evaluated on mainstream LVLMs.

Problem

LVLMs hallucinate object presence, and existing benchmarks detect hallucination without formally characterizing or quantifying how co-occurrence-driven spurious correlations cause it.

Approach

Model co-occurrence bias as a confounder in a structural causal model and apply visual content intervention, replacing high-co-occurrence objects with low-co-occurrence ones via T2I inpainting to create counterfactual images, then quantify causal effects with ACE-based and DCS-based metrics.

Key results

• Kimi-VL-A3B has the highest AAC at 10.2%, while LLaVA-NEXT-8B achieves the best CHR at 6.8%, and InternVL2.5-8B has the worst CHR at 29.3%.
• Most models show high CAC; Qwen2.5-VL-7B has the lowest CAC at 1.8% but a CHR of 27.3%.
• Synthetic counterfactual images show CLIP Score 22.4 for target and 27.5 for counterfactual objects versus 26.5 and 20.6 for originals.

Surprising findings

• Newer models such as Qwen2.5-VL may be more susceptible to spurious correlations than earlier models like LLaVA-NEXT despite better general benchmark performance.
• Closed-source models (GPT-4o, Gemini1.5-pro) do not significantly outperform open-source models on the causal metrics.

Limitations

• Inpainting is only an approximation of the theoretical do-intervention and may introduce visual artifacts.
• Dataset is small (757 original images, 2144 total images) and relies on manual filtering.
• Counterfactual samples depend on proprietary Gemini for object selection and description.

Experimental setup

Datasets: MSCOCO validation set

Benchmarks: Causal-HalBench POPE CHAIR

Metrics: accuracy CAC AAC CHR CHAIR-S CHAIR-I CLIP Score

Baselines: POPE CHAIR BEAF

Builds on: POPE BEAF CHAIR FLUX-controlnet inpainting Segment Anything Model

Closest papers by meaning

• Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.93)
• Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations (CVPR 2026 · similarity 0.93)
• Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (ICLR 2024 · similarity 0.92)
• Multi-Object Hallucination in Vision Language Models (NeurIPS 2024 · similarity 0.92)
• Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination (CVPR 2026 · similarity 0.92)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.92)