Research map / Vision-language and multimodal / Multimodal LLMs
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Visual Contrastive Decoding (VCD), a training-free decoding method that contrasts output distributions from original and Gaussian-noise-distorted images to reduce object hallucination in LVLMs.
Problem
Object hallucination in large vision-language models, where generated text includes objects absent from the image, driven by statistical bias and language priors.
Approach
Amplify hallucination-prone behavior by decoding from a Gaussian-noise-distorted image and contrast its logits with those from the original image at each decoding step, with an adaptive plausibility constraint to keep only confident tokens; no training or external models needed.
Key results
- On POPE, VCD improves LLaVA-1.5 random-setting MSCOCO F1 from 81.33 to 87.16 (up to +7.4 F1 across settings).
- On MME hallucination subset, LLaVA-1.5 total score rises from 565.33 to 604.66 with VCD.
- GPT-4V-judged LLaVA-Bench accuracy for Qwen-VL improves from 4.76 to 6.69 with VCD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Visual Contrastive Decoding | LLaVA1.5 · POPE MSCOCO Random | F1 score: 81.33_{(\pm0.41)} -> 87.16_{(\pm0.41)} | Helped |
| Apply Visual Contrastive Decoding | LLaVA1.5 · POPE MSCOCO Popular | F1 score: 80.06_{(\pm0.05)} -> 85.06_{(\pm0.37)} | Helped |
| Apply Visual Contrastive Decoding | LLaVA1.5 · POPE MSCOCO Adversarial | F1 score: 77.57_{(\pm0.57)} -> 81.33_{(\pm0.34)} | Helped |
| Apply Visual Contrastive Decoding | Qwen-VL · POPE A-OKVQA Random | F1 score: 85.59_{(\pm0.53)} -> 89.01_{(\pm0.16)} | Helped |
| Apply Visual Contrastive Decoding | InstructBLIP · POPE GQA Random | F1 score: 80.56_{(\pm0.18)} -> 84.16_{(\pm0.01)} | Helped |
| Apply Visual Contrastive Decoding | LLaVA1.5 · MME hallucination subset | Total Scores: 565.33_{(\pm32.92)} -> 604.66_{(\pm18.76)} | Helped |
| Apply Visual Contrastive Decoding | Qwen-VL · MME hallucination subset | Total Scores: 587.33_{(\pm31.06)} -> 596.67_{(\pm11.61)} | Helped |
| Apply Visual Contrastive Decoding | InstructBLIP · MME hallucination subset | Total Scores: 380.33_{(\pm40.20)} -> 447.67_{(\pm13.36)} | Helped |
| Apply Visual Contrastive Decoding | LLaVA-1.5 · LLaVA-Bench | Accuracy: 3.23 -> 4.15 | Helped |
| Apply Visual Contrastive Decoding | InstructBLIP · LLaVA-Bench | Accuracy: 3.84 -> 4.23 | Helped |
| Apply Visual Contrastive Decoding | Qwen-VL · LLaVA-Bench | Accuracy: 4.76 -> 6.69 | Helped |
| Use T=999 instead of T=500 noise steps (ablation) | VCD on LLaVA-1.5 · MME hallucination subset | Total Scores: 591.67_{\pm 36.06} -> 557.78_{\pm 1.92} | Hurt |
Surprising findings
- Increasing model scale from 7B to 13B does not resolve hallucination; 13B LLaVA-1.5 and InstructBLIP perform comparably to their 7B versions on POPE.
- Ablating the adaptive plausibility constraint (beta=0) yields suboptimal results, showing that unconstrained contrast penalizes valid tokens.
Limitations
- Uses only simple Gaussian noise to introduce visual uncertainty; finer distortions such as object-level blurring not explored.
- Limited to image-text LVLMs; video understanding not covered.
- Hyperparameters (alpha, beta, T) fixed in main experiments and not tuned per model, so results may be suboptimal.
Experimental setup
Datasets: MSCOCO A-OKVQA GQA LLaVA-Bench
Benchmarks: POPE MME LLaVA-Bench
Metrics: accuracy precision recall F1 score MME total score accuracy (GPT-4V judged) detailedness
Baselines: Regular decoding POPE MME
Builds on: contrastive decoding classifier-free guidance adaptive plausibility constraints POPE
Closest papers by meaning
- SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.96)
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.94)
- Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention (CVPR 2025 · similarity 0.94)
- Mitigating Object Hallucination in Large Vision-Language Models via False Discovery Controlled Visual Data Splitting (NeurIPS 2026 · similarity 0.94)
- Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations (ICLR 2025 · similarity 0.94)
- ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large Language Models (CVPR 2025 · similarity 0.94)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.