Research map / Vision-language and multimodal / Multimodal LLMs

The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that contrastive decoding methods (VCD, ICD, SID) gain on POPE mainly from a unidirectional shift toward "Yes" and from the adaptive plausibility constraint collapsing sampling into greedy search, not from mitigating hallucination.

Problem

Contrastive decoding methods are widely reported to reduce object hallucination in MLLMs, mainly via POPE gains; the paper asks whether these gains reflect genuine hallucination mitigation.

Approach

Designs simple spurious improvement methods (prompt-based adjustment and output-layer modification that bias toward "Yes", and the adaptive plausibility constraint applied alone under sampling) that match contrastive decoding's POPE gains without addressing hallucination, exposing the confounds.

Key results

• Prompt-based adjustment and output-layer modification, which only bias toward 'Yes', reach POPE COCO-Random gains comparable to VCD and SID (e.g., OLM 89.6 vs VCD 88.6 on LLaVA-v1.5-7B).
• Applying the adaptive plausibility constraint alone under sampling yields about 1.5-2% accuracy gains on POPE, roughly matching VCD/SID/ICD.
• On generative benchmarks (CHAIR, NoCaps, LLaVA-Bench) contrastive decoding gives no improvement under greedy decoding and at best marginal gains under sampling.

Measured effects reported in the paper

Surprising findings

• Simple forced 'Yes'-biasing modifications unrelated to hallucination match contrastive decoding's POPE gains.
• Contrastive decoding under greedy search degrades MME and CHAIR-S, while the adaptive plausibility constraint alone under sampling is the largest source of gains on other benchmarks.

Limitations

• Analysis centers on POPE and a few MLLMs (LLaVA-v1.5, QwenVL-Chat), limiting generality.
• Spurious methods are shown to match gains but the paper does not propose a new validated hallucination mitigation method.

Experimental setup

Datasets: MSCOCO GQA AOKVQA MME CHAIR NoCaps LLaVA-Bench

Benchmarks: POPE MME CHAIR NoCaps LLaVA-Bench AOKVQA

Metrics: accuracy F1-score Yes ratio CHAIR-S CHAIR-I MME perception MME cognition

Baselines: VCD ICD SID Greedy decoding Direct sampling

Builds on: Visual Contrastive Decoding Instruction Contrastive Decoding Self-Introspective Decoding POPE

Closest papers by meaning

• ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models (AAAI 2025 · similarity 0.94)
• MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation (ICLR 2025 · similarity 0.94)
• Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization (NeurIPS 2024 · similarity 0.94)
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.93)
• SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.93)
• Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.93)