Research map / Vision-language and multimodal / Multimodal LLMs

Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FarSight, a training-free plug-and-play causal-mask modification (attention registers in the upper triangle plus a diminishing masking rate) that reduces hallucinations in image and video MLLMs.

Problem

MLLMs hallucinate, including initial hallucinations and snowball hallucinations that compound earlier errors; attention collapse onto outlier tokens and RoPE long-term decay limit vision-to-text information propagation.

Approach

Replace the standard causal mask with an upper-triangular attention-register matrix holding linearly decaying scores (softmax absorbs excess attention, then future positions are zeroed), plus a decay rate that encodes absolute position, so vision and text tokens keep propagating information without any retraining.

Key results

• On LLaVA-1.5, FarSight lowers CHAIR_S from 48.0 to 41.6 and raises POPE-R from 87.0 to 90.5 and POPE-P from 82.8 to 86.1.
• Adding FarSight to Video-LLaVA reduces CHAIR_S from 50.2 to 44.8 and improves MSVD-QA accuracy from 64.8 to 66.2 (reported as +1.4).
• FarSight improves MMBench and LLaVA^W for LLaVA-1.5 by +1.7 and +2.2 points, respectively.

Measured effects reported in the paper

Surprising findings

• Hard -infinity masking in the upper triangle, a common choice, performs worse than the register design, as reported in the ablation.
• Some baseline decoding methods (e.g., ICD, VCD) reduce accuracy on certain general VQA benchmarks, such as VizWiz for ICD and VCD.

Limitations

• Relies on a hand-set decay rate (sigma from log base 1024 of seq 256) chosen empirically; sensitivity shown only on a few settings.
• Evaluation is on a limited set of MLLMs and benchmarks; hallucination analysis depends on GPT-4o-labeled snowball categorization.
• Gains on some metrics are modest (e.g., +0.5 POPE-A on Video-LLaVA, +0.7 CHAIR_I on LLaVA-1.5).

Experimental setup

Datasets: CHAIR POPE MSCOCO

Benchmarks: MMBench LLaVA-Bench (LLaVA^W) MM-Vet VizWiz ScienceQA CHAIR POPE MSVD-QA MSRVTT-QA ActivityNet-QA Video-Based Text Generation Benchmark

Metrics: accuracy CHAIR_S CHAIR_I POPE F1-style accuracy (R/P/A) perplexity GPT score

Baselines: ICD VCD OPERA EDVT FixVPE

Builds on: RoPE OPERA EDVT VCD ALiBi

Closest papers by meaning

• Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models (ICLR 2025 · similarity 0.94)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.93)
• Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.93)
• Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.92)
• Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection (CVPR 2026 · similarity 0.92)