Research map / Vision-language and multimodal / Multimodal LLMs

Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GIFT, an inference-time method that builds a visual saliency map from positive "gaze shifts" in query-to-visual attention and uses it to amplify both visual and query attention during decoding to reduce VLM hallucination.

Problem

VLMs hallucinate by over-relying on language priors and under-using visual inputs; existing attention-steering methods amplify visual attention without addressing visual attention sinks or cross-modal fusion balance.

Approach

Compute a holistic visual saliency map from positive changes in query-to-visual attention over information-rich query tokens (gaze shifts) at a peak layer, then at decoding time amplify attention to salient visual tokens and proportionally scale query-token attention to keep cross-modal balance.

Key results

• GIFT reduces CHAIR C_s on LLaVA-1.5 7B from 50.2 (greedy) to 39.8, a relative gain up to 20.7% on CHAIR across models.
• On MMHal-Bench, GIFT lowers hallucination rate on LLaVA-1.5 7B from 65.2 to 57.3 and raises score from 2.22 to 2.48.
• GIFT runs at 1.13x greedy decoding latency versus 11.10x for VAR and 1.99x for VCD.

Surprising findings

• Qwen3-VL 8B achieves the best POPE and MMHal-Bench scores yet shows higher CHAIR scores, which the authors attribute to CHAIR's string-matching penalizing detailed outputs.

Limitations

• Relies on the query to identify relevant visual regions; inaccurate saliency when answering does not depend on the image or query is unrelated to image.
• Inference still about 13% slower than greedy decoding.
• Hyperparameters tuned on pseudo-validation splits of POPE and MME; not all decoding steps need visual attention.

Experimental setup

Datasets: TextVQA MathVista MSCOCO 2014

Benchmarks: CHAIR POPE MMHal-Bench MME SEED-Bench AMBER MMBench

Metrics: CHAIR C_s CHAIR C_i F1 accuracy hallucination rate informativeness score latency

Baselines: Greedy decoding VAF Rel-Attn (MLLMs_know) VAR VCD

Builds on: Visual Amplification Fusion (VAF) VCD visual attention redistribution (VAR) attention steering

Closest papers by meaning

• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.93)
• ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large Language Models (CVPR 2025 · similarity 0.93)
• Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation (AAAI 2026 · similarity 0.93)
• Cross-Modal Attention Calibration for LVLM Hallucination Mitigation (CVPR 2026 · similarity 0.92)
• Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation (ICLR 2026 · similarity 0.91)
• VASparse: Towards Efficient Visual Hallucination Mitigation via Visual-Aware Token Sparsification (CVPR 2025 · similarity 0.91)