Research map / Vision-language and multimodal / Multimodal LLMs
Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CausalMM, a training-free causal inference decoding framework that treats modality priors as confounders and applies back-door adjustment and counterfactual attention interventions at visual and language attention levels to reduce multimodal hallucinations.
Problem
MLLMs suffer from hallucinations due to visual and language priors biasing the attention mechanism, and existing decoding methods rely on statistical correlations rather than causal relations between attention and output.
Approach
Builds a structural causal model of MLLMs with modality priors as confounders, then estimates causal effects of visual and language attention via counterfactual attention interventions (random, uniform, reversed, shuffled) and back-door adjustment, combined into a contrastive-style decoding rule that is plug-and-play.
Key results
- On POPE, the multimodal CausalMM variant reaches 88.93 accuracy on MSCOCO random versus 83.53 for regular decoding with LLaVA-1.5, with an average metric gain reported as 5.37%.
- Abstract reports a maximum score improvement of 65.3% on six VLind-Bench indicators and a 164-point gain on MME.
- With GPT-4o as judge on LLaVA-Bench, the multimodal variant scores 85.0 overall versus 84.7 for regular decoding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multimodal collaborative counterfactual attention | LLaVA-1.5 · POPE MSCOCO Adversarial | accuracy: 78.63 -> 83.70 | Helped |
| Add multimodal collaborative counterfactual attention | LLaVA-1.5 · POPE GQA Popular | accuracy: 77.86 -> 82.36 | Helped |
| Add multimodal collaborative counterfactual attention | LLaVA-1.5 · POPE A-OKVQA Random | accuracy: 84.03 -> 88.93 | Helped |
| Add multimodal collaborative counterfactual attention | LLaVA-1.5 · POPE MSCOCO Random | accuracy: 89.20 -> 88.93 | Hurt |
| Add multimodal collaborative counterfactual attention | Qwen2-VL · MME perception OCR | score: 147.50 -> 170.00 | Helped |
| Add multimodal collaborative counterfactual attention | LLaVA-1.5 · LLaVA-Bench GPT-4o evaluation | All: 84.7 -> 85.0 | Helped |
| Use language-only counterfactual attention instead of regular decoding | LLaVA-1.5 · POPE A-OKVQA Popular | accuracy: 80.23 -> 85.96 | Helped |
| Use random attention as the counterfactual anchor (ablation) | CausalMM · Model performance (averaged results) | performance | Helped |
| Intervene in shallow-to-middle LLM layers rather than other depths (ablation) | CausalMM language attention · Model performance | performance | Helped |
Surprising findings
- Random attention as the counterfactual anchor gave the largest improvement, which the authors attribute to its closeness to average attention.
- Language-only intervention on a large LLM is less sensitive to attention perturbation than visual encoder intervention.
Limitations
- Gains are bounded by the performance of the vision encoder and LLM backbone; high-level semantics are not improved much.
- Evaluation of open-ended quality relies on a single GPT-4o judge, and improvements on some MME subsets are inconsistent (e.g., position stays flat).
Experimental setup
Datasets: VLind-Bench POPE MME LLaVA-Bench
Benchmarks: VLind-Bench POPE MME LLaVA-Bench
Metrics: accuracy precision recall F1 score VLind-Bench pipeline score MME score GPT-4o rating
Baselines: Regular decoding VCD OPERA DoLa AGLA
Builds on: Visual Contrastive Decoding OPERA structural causal model back-door adjustment
Closest papers by meaning
- The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio (NeurIPS 2025 · similarity 0.93)
- CausalLens: Sensitivity-Guided Multi-Head Causal Intervention for Hallucination Mitigation in Large Vision-Language Models (CVPR 2026 · similarity 0.93)
- Robust Multimodal Large Language Models Against Modality Conflict (ICML 2025 · similarity 0.93)
- Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection (CVPR 2026 · similarity 0.92)
- Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.92)
- AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.