Research map / Vision-language and multimodal / Multimodal LLMs

Hallucination Begins Where Saliency Drops

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LVLMs-Saliency (gradient-attention fusion) to diagnose hallucinations, and two inference-time methods, SGRS (saliency-based rejection sampling) and LocoRE (attention reinforcement to recent output tokens), that reduce hallucinations in LVLMs.

Problem

Hallucinations in large vision-language models are poorly explained by forward-pass attention maps, which ignore how token influence propagates via gradients; existing attention-based diagnostics do not reliably separate hallucinated from correct tokens.

Approach

Define a token-level saliency as the element-wise product of attention and its gradient, observe that hallucinated tokens have low saliency to prior output tokens, then reject low-saliency candidates at decode time (SGRS) and boost attention from the current query to recent output tokens with a distance-aware gain (LocoRE).

Key results

• SGRS+LocoRE on LLaVA-1.5-7B reduces CHAIR_S from 48.0 to 35.6 and CHAIR_I from 13.9 to 8.2, and raises POPE F1 from 85.4 to 87.0.
• LocoRE on Qwen2-VL-7B lowers CHAIR_S from 25.0 to 23.5 and raises MME-style hallucination-benchmark POPE-A from 87.6 to 88.2.
• Hallucinated tokens show mean saliency 0.193 vs 0.472 for correct tokens on LLaVA-v1.5-7B (reported in rebuttal experiments).

Surprising findings

• Authors report that high-confidence incorrect tokens can still show high saliency, a failure case of the low-saliency hypothesis.
• Prompt saliency differences between correct and hallucinated tokens do not significantly affect predictive accuracy according to a 500-sample analysis.

Limitations

• SGRS requires a backward pass, adding 30-40% per-token latency.
• High-confidence incorrect tokens can exhibit high saliency, violating the low-saliency pattern.
• Hallucination labels in the analysis rely on human annotation of a limited token sample (~12,000 tokens).

Experimental setup

Datasets: POPE CHAIR MME LLaVA-Bench MM-Vet VizWiz ScienceQA

Benchmarks: POPE CHAIR MME LLaVA-Bench MM-Vet VizWiz ScienceQA

Metrics: F1 accuracy CHAIR_S CHAIR_I recall MME total score POPE recall

Baselines: Beam Search DoLa VCD OPERA DOPRA HALC CCA-LLaVA RITUAL EAH SID TAME Vissink

Builds on: OPERA VCD EAH TAME Label Words information flow

Closest papers by meaning

• Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy (ICML 2026 · similarity 0.93)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.92)
• Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation (AAAI 2026 · similarity 0.92)
• AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM (CVPR 2026 · similarity 0.91)
• Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination (CVPR 2026 · similarity 0.91)
• Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating (ICML 2026 · similarity 0.91)