Research map / Vision-language and multimodal / Multimodal LLMs

The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models Via Visual Information Steering

ICML 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VISTA, a training-free inference-time method that steers residual-stream activations with a per-image visual steering vector and mixes in early-layer logits to reduce object hallucination in LVLMs.

Problem

LVLMs hallucinate syntactically coherent but visually ungrounded content; the internal dynamics of how visual information is lost during generation are poorly understood.

Approach

Token-logit ranking analysis via logit lens reveals gradual visual information loss, early excitation of semantic tokens in late layers, and hidden genuine tokens; VISTA counters these with a per-image visual steering vector (positive minus no-image context residuals) injected into all layers and a Self-Logits Augmentation that averages logits from the last w layers before the final one.

Key results

• On CHAIR with greedy decoding, VISTA reduces LLaVA-1.5 CHAIR_S from 46.4 to 20.4 and CHAIR_I from 12.1 to 6.9, roughly 40% relative improvement over vanilla.
• On POPE under nucleus sampling, VISTA raises LLaVA-1.5 average accuracy from 81.26 to 85.35 and MiniGPT-4 from 60.56 to 66.96.
• VISTA's latency is 36.32 ms/token versus 58.34 for VCD and 57.78 for PAI on LLaVA-1.5 (greedy), though it is slower than vanilla greedy at 28.54 ms/token.

Measured effects reported in the paper

Surprising findings

• Hidden genuine tokens not decoded in the output still keep relatively high logit rankings (around 5K of 32K vocabulary) during generation.
• Final-layer logits prioritize functional tokens (stopwords) while semantically meaningful tokens peak in the penultimate layers (early excitation).

Limitations

• Optimal hyperparameters (lambda, gamma, window size) vary across architectures and require tuning on a validation set.
• Effectiveness of VSV depends on the quality of visual cues from the vision encoder; weak encoders may see reduced benefits.
• Evaluated only on single-round tasks; interactive visual dialogue not addressed.

Experimental setup

Datasets: MS COCO

Benchmarks: CHAIR POPE MMHal-Bench MME

Metrics: CHAIR_S CHAIR_I accuracy F1 MMHal score MME perception+cognition score latency throughput

Baselines: Vanilla decoding DoLa VCD OPERA PAI

Builds on: DoLa Visual Contrastive Decoding Prompt-to-Attention / PAI steering vectors logit lens

Closest papers by meaning

• Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.93)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.93)
• Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision-Language Models (ICML 2026 · similarity 0.93)
• Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models (ICLR 2025 · similarity 0.93)
• Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy (ICML 2026 · similarity 0.93)
• Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating (ICML 2026 · similarity 0.93)