Research map / Vision-language and multimodal / Multimodal LLMs
Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CMI-VLD, a decoding strategy that maximizes conditional pointwise mutual information between image and generated text via calibrated text sampling plus a learned visual token purifier, reducing hallucinations in LVLMs.
Problem
LVLMs hallucinate because generation over-relies on language priors and weakly depends on the input image, leading to low mutual dependency between image and response.
Approach
Formulates hallucination mitigation as bi-level maximization of C-PMI: a calibrated contrastive text distribution (with and without image) for the inner problem, and a learned Gumbel-Softmax visual token purifier that retains image tokens most relevant to the current context and generated response for the outer problem.
Key results
- On CHAIR with greedy decoding, CMI-VLD reduces C_S on LLaVA-1.5 from 45.0 (greedy) to 29.9 and on LLaVA-NeXT to 27.2.
- On MME, CMI-VLD scores 1481.17 versus 1465.11 for greedy on LLaVA-1.5.
- Reports a 15.89% relative SHR improvement over competitive baselines for LLaVA-1.5 on the GPT-4o assisted benchmark.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add calibrated distribution sampling and visual token refinement together (ablation) | CMI-VLD · MSCOCO hallucination evaluation | hallucination performance | Helped |
| Add calibrated distribution sampling (ablation) | CMI-VLD · MSCOCO hallucination evaluation | hallucination performance | Helped |
| Add visual token refinement (ablation) | CMI-VLD · MSCOCO hallucination evaluation | hallucination performance | Helped |
| Include attention scores in purifier training (positive alpha rather than alpha=0) (ablation) | CMI-VLD visual purifier · MSCOCO CHAIR | hallucination performance | Helped |
| Set alpha to 1×10^2 (ablation) | CMI-VLD visual purifier · MSCOCO CHAIR | hallucination performance | Helped |
| Use positive calibration strength lambda instead of lambda=0 (ablation) | CMI-VLD · MSCOCO CHAIR | C_S and C_I | Helped |
| Set calibration strength lambda to 0.5 (ablation) | CMI-VLD · MSCOCO CHAIR | hallucination performance | Helped |
| Set visual token retention ratio gamma to 80% (ablation) | CMI-VLD on LLaVA-1.5 · MSCOCO CHAIR | hallucination performance | Helped |
| Replace the learned purifier with manual per-step token selection (ablation) | CMI-VLD · MSCOCO CHAIR, four LVLMs | C_S: 30.0 vs 29.9; 40.4 vs 43.2; 36.2 vs 30.6; 26.6 vs 27.2 | Mixed |
| Replace the learned purifier with manual per-step token selection (ablation) | CMI-VLD · Decoding efficiency | inference latency | Hurt |
Surprising findings
- On InstructBLIP CHAIR, CMI-VLD (C_S 43.2) underperforms VCD-free greedy-style baselines such as VCD (46.6) on C_S in sampling-free comparisons, and ICD/VCD beat it on some POPE adversarial metrics.
- Authors note the method gives little gain on POPE because responses hinge on the first Yes/No token.
Limitations
- Purifier training adds computational overhead, and efficiency gains diminish for very long responses.
- Performance is sensitive to the retaining ratio, which is fixed across decoding steps although optimal ratio may vary.
- Gains on POPE are limited since the method adjusts decoding over the full generation.
Experimental setup
Datasets: ShareGPT4V MSCOCO Visual Genome
Benchmarks: CHAIR GPT-4o assisted SHR benchmark POPE MME MMBench
Metrics: C_S C_I SHR accuracy F1 score TPS inference latency FLOPs
Baselines: Sampling Greedy VTI VCD ICD HALC OPERA SID VASparse
Builds on: visual contrastive decoding SID VTI adaptive plausibility truncation
Closest papers by meaning
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.95)
- Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.95)
- Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models (ICLR 2025 · similarity 0.94)
- Multi-Modal Hallucination Control by Visual Information Grounding (CVPR 2024 · similarity 0.94)
- MoLE:Decoding by Mixture of Layer Experts Alleviates Hallucination in Large Vision-Language Models (AAAI 2025 · similarity 0.94)
- Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.94)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.