Research map / Vision-language and multimodal / Multimodal LLMs

Cross-Modal Attention Calibration for LVLM Hallucination Mitigation

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CMAC, a training-free decoding method for LVLMs that combines Inter-Modality Decoding (masking value vectors at high cross-modal attention for contrastive decoding) with Cross-Modal Position Calibration (scaling image-token RoPE position indices) to reduce hallucinations.

Problem

LVLMs hallucinate objects in generated text; existing contrastive decoding methods only distort unimodal inputs and ignore spurious inter-modality correlations and RoPE-induced position bias toward later image tokens.

Approach

Instead of distorting whole images or text, mask value vectors tied to the largest text-to-image attention weights to build a hallucination-prone distribution for contrastive decoding, and linearly shrink image-token position indices in the cross-modal attention so the head of the image is not under-attended.

Key results

• On POPE (sampling, random), LLaVA-1.5 accuracy rises from 83.49 baseline to 89.10 with CMAC, versus 87.73 for the best prior method PAI.
• On CHAIR with nucleus sampling, CMAC lowers LLaVA-1.5 CHAIR_s from 17.8 to 12.7 and CHAIR_i from 55.6 to 47.0.
• On the MME hallucination subset, LLaVA1.5 total score improves from 566.67 to 612.49.

Measured effects reported in the paper

Surprising findings

• Global position calibration (GPC) on all tokens yields higher POPE accuracy (86.57) than CMAC but substantially increases CHAIR_i (61.6), suggesting it harms caption length control.
• Attention masking (zeroing high-attention weights) performs worse than value masking, indicating altering attention weights misestimates image-region hallucination.

Limitations

• Attention magnitude may not capture causal relevance between text and visual tokens, so the masking selection is heuristic.
• Effectiveness is bounded by the visual encoder's grounding capability since CMAC does not improve visual representations.
• Less effective on numerical calculation and text translation MME subtasks, which depend on language reasoning.

Experimental setup

Datasets: MS-COCO A-OKVQA GQA

Benchmarks: POPE CHAIR MME MMBench

Metrics: accuracy F1 CHAIR_i CHAIR_s recall MME score GPT-4o score

Baselines: VCD ICD OPERA PAI Baseline standard decoding

Builds on: Contrastive Decoding VCD ICD OPERA PAI

Closest papers by meaning

• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.93)
• One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination (CVPR 2026 · similarity 0.92)
• Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation (AAAI 2026 · similarity 0.92)
• Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation (ICML 2026 · similarity 0.92)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.92)
• Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality (ICLR 2025 · similarity 0.91)