Research map / Vision-language and multimodal / Multimodal LLMs

Mitigating Hallucinations in Large Vision-Language Models by Adaptively Constraining Information Flow

AAAI 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AdaVIB, which fine-tunes only the vision-language projector of LVLMs with a variational information bottleneck whose noise level adapts to the entropy of the visual-token/word-embedding similarity distribution, to reduce object hallucination.

Problem

Object hallucination in large vision-language models, attributed to overconfidence in irrelevant visual features when soft visual tokens are projected into the LLM word embedding space.

Approach

Applies the Variational Information Bottleneck to the vision-language projector to inject stochastic noise that constrains irrelevant visual information, and makes the compression weight beta adaptive per sample via the entropy of the visual-token-to-word-embedding similarity distribution.

Key results

• On MSCOCO CHAIR_S, AdaVIB reaches 16.2 (MiniGPT-4) and 18.4 (LLaVA-1.5) versus 27.1 and 47.8 for the original models and 19.3 and 26.7 for fine-tuning.
• On POPE Popular split, AdaVIB achieves 71.1 accuracy on MiniGPT-4 and 86.3 on LLaVA-1.5, improving over LURE by 8.1% and 7.5% on average-reported terms.
• Removing the reparameterization strategy raises CHAIR_S by over 15% on average, and removing adaptive beta raises it by over 5% on average.

Measured effects reported in the paper

Surprising findings

• Fine-tuning the projector can increase hallucination relative to the frozen projector, with sharper similarity distributions correlating with hallucinated outputs.
• Dropout regularization on the projector reduces hallucinations less than AdaVIB, though it also helps over plain fine-tuning.

Limitations

• Evaluation limited to two LVLM architectures and two object hallucination benchmarks.
• Beta is sensitive: large beta degrades performance and very small beta loses the compression benefit.
• Gradients do not flow through the adaptive beta, a design choice without theoretical justification in the text.

Experimental setup

Datasets: MSCOCO 2014 LLaVA-150k

Benchmarks: CHAIR (MSCOCO 2014) POPE (Random, Popular, Adversarial)

Metrics: CHAIR_S CHAIR_I Accuracy F1

Baselines: Chain-of-Thought DOLA Teacher (long-form caption generator) VCD LURE Original model Fine-tuned (FT) projector Dropout

Builds on: Variational Information Bottleneck Information Bottleneck LLaVA-1.5 MiniGPT-4 reparameterization trick

Closest papers by meaning

• Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations (ICLR 2025 · similarity 0.93)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
• On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models (NeurIPS 2025 · similarity 0.93)
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.93)
• Image Token Matters: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing (NeurIPS 2025 · similarity 0.93)
• HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models (ICML 2026 · similarity 0.93)