Research map / Vision-language and multimodal / Multimodal LLMs

On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes identifying epistemically uncertain visual tokens in a vision encoder via PGD adversarial feature deviation and masking them in mid-layer self-attention, a training-free VE-only method that reduces object hallucination in LVLMs.

Problem

Object hallucination in LVLMs (describing objects absent from the image) is linked to visual token uncertainty in the vision encoder, but estimating epistemic uncertainty via MC dropout is costly and prior mitigation methods largely ignore the vision encoder.

Approach

Use PGD adversarial perturbations to approximate per-token epistemic uncertainty (feature deviation in early VE layers, justified by a local Gaussianity lemma and entropy-bound theorem), then hard-mask the uncertain tokens' outputs in the self-attention of intermediate VE layers, without any training.

Key results

• On LLaVA-1.5-7B with greedy decoding, CHAIR_s drops from 47.4 to 29.2 and CHAIR_i from 12.2 to 9.3 with the proposed masking.
• Adversarial-attack uncertainty estimation is about 5.1x faster than MC dropout (2.43 s vs 12.4 s per example).
• Combined with OPERA on AMBER (LLaVA-1.5-7B), the AMBER score rises from 89.1 to 91.8 and CHAIR drops 21.6%.

Measured effects reported in the paper

Surprising findings

• Soft masking of uncertain tokens (attenuation by 0.1 or 0.2) is less effective than hard masking in self-attention.
• Adversarial attacks on the vision encoder degrade LVLM outputs, with hallucination severity rising with attack strength.

Limitations

• Gains on MiniGPT-4 are smaller, attributed to the Q-Former limiting the effect of modifying the vision encoder.
• Per-baseline-method sigma_th values are tuned, requiring some hyperparameter selection.
• The theoretical argument relies on local Gaussianity and small-perturbation approximations; the covariance was found to be low-rank, so the trace is used in place of the determinant.

Experimental setup

Datasets: MS-COCO

Benchmarks: CHAIR POPE AMBER

Metrics: CHAIR_s CHAIR_i F1 accuracy AMBER score Hal Cog Precision Recall

Baselines: Greedy OPERA VCD PAI Devils

Builds on: PGD MC dropout OPERA VCD PAI Devils in the middle layers

Closest papers by meaning

• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
• Mitigating Object Hallucination in Large Vision-Language Models via False Discovery Controlled Visual Data Splitting (NeurIPS 2026 · similarity 0.93)
• Mitigating Hallucinations in Large Vision-Language Models by Adaptively Constraining Information Flow (AAAI 2025 · similarity 0.93)
• Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations (ICLR 2025 · similarity 0.93)
• Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.93)
• Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.93)