Research map / Vision-language and multimodal / Multimodal LLMs
Multi-Modal Hallucination Control by Visual Information Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M3ID, a training-free inference-time decoding that amplifies visual-prompt dependence (mutual information) to reduce object hallucinations in VLMs, optionally paired with DPO on self-generated preference pairs.
Problem
Generative vision-language models hallucinate objects not grounded in the image; the influence of the visual prompt fades as more tokens are generated (conditioning dilution), so models rely on the language prior.
Approach
Model the conditioned log-probs as a fading-memory interpolation with the unconditioned model and correct decoding by adding a time-decaying multiple of (l_c - l_u), i.e., a pointwise mutual information term, gated by a confidence threshold to avoid penalizing obvious tokens; optionally train with DPO using M3ID captions as positives and unconditioned continuations as negatives.
Key results
- For LLaVA-13B, M3ID reduces CHAIRi from 7.4 to 5.5 and CHAIRs from 18.5 to 13.2 while keeping Cover near 54%.
- On POPE, LLaVA-13B M3ID raises overall accuracy from 63.8 to 77.5 (about 21% relative improvement); M3ID+DPO reaches 79.2.
- LLaVA-7B M3ID+DPO achieves CHAIRi 5.7 and CHAIRs 13.5 versus 8.1 and 17.5 for the base model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply M3ID decoding | LLaVA-7B · MS COCO val captioning | CHAIRi: 8.1 -> 5.9 | Helped |
| apply M3ID decoding | LLaVA-13B · POPE All | accuracy: 63.8 -> 77.5 | Helped |
| add DPO to M3ID | LLaVA-13B · POPE All | accuracy: 77.5 -> 79.2 | Helped |
| set λ to 0.001 instead of 0.02 (ablation) | LLaVA-7B M3ID · MS COCO val captioning | CHAIRi: 5.8 -> 7.3 | Hurt |
| set λ to 0.03 instead of 0.02 (ablation) | LLaVA-7B M3ID · MS COCO val captioning | CHAIRi: 5.8 -> 6.2 | Hurt |
| set λ to 0.1 instead of 0.02 (ablation) | LLaVA-7B M3ID · MS COCO val captioning | CHAIRi: 5.8 -> 7.2 | Hurt |
| set α to 0.5 instead of 0.3 (ablation) | LLaVA-7B M3ID · MS COCO val captioning | CHAIRi: 5.8 -> 6.1 | Hurt |
| set α to 0.1 instead of 0.3 (ablation) | LLaVA-7B M3ID · MS COCO val captioning | CHAIRi: 5.8 -> 6.9 | Hurt |
| set α to 0.01 instead of 0.3 (ablation) | LLaVA-7B M3ID · MS COCO val captioning | CHAIRi: 5.8 -> 6.9 | Hurt |
| add context-pressure correction (ablation) | LLaVA-7B M3ID · POPE All | accuracy: 77.5 -> 65.5 | Hurt |
| add conditioning-dilution amplification (ablation) | LLaVA-7B decoding · POPE All | accuracy: 64.9 -> 77.5 | Helped |
| add context-pressure correction (ablation) | LLaVA-13B M3ID · POPE All | accuracy: 70.3 -> 64.9 | Hurt |
Surprising findings
- Strong language-prior penalization causes "overcompensation": captions omit obvious tokens predictable from text context (e.g., the man or van), and excessive correction leads to degenerate, non-English tokens.
- The base LLaVA yes-answer bias on POPE (about 84%) is largely reduced by M3ID, which mainly improves accuracy through answer-bias correction.
Limitations
- Requires two forward passes at inference (conditioned and unconditioned), doubling compute or memory.
- Overcompensation can suppress predictable but correct tokens, lowering Cover and hurting fluency at high alpha or lambda.
- Hyperparameters (alpha, lambda) require tuning on held-out data; POPE needs an offset t0 for prompt length.
Experimental setup
Datasets: MS COCO LCS-558K LLaVA-Instruct-80K
Benchmarks: MS COCO CHAIR POPE
Metrics: CHAIRi CHAIRs Cover POPE accuracy Yes ratio
Baselines: PMI decoding Contrastive Decoding LLaVA-RLHF Robust mPLUG-Owl LURE InstructBLIP
Builds on: Contrastive Decoding PMI decoding DPO LLaVA
Closest papers by meaning
- Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.94)
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.94)
- Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.94)
- SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.93)
- Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models (ICLR 2025 · similarity 0.92)
- MoLE:Decoding by Mixture of Layer Experts Alleviates Hallucination in Large Vision-Language Models (AAAI 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.