Research map / Vision-language and multimodal / Multimodal LLMs

Breaking the Illusion: When Positive Meets Negative in Multimodal Decoding

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Positive-and-Negative Decoding (PND), a training-free dual-path contrastive decoding method that amplifies attention-salient visual evidence and degrades core object regions via DDPM noising, to reduce object hallucination in VLMs.

Problem

Object hallucination in vision-language models, attributed to over-reliance on language priors and under-weighting of visual evidence in deeper decoding layers (both object invention and omission).

Approach

Combines a positive path that multiplicatively amplifies multi-layer cross-modal attention salience maps with a negative path that replaces consensus-attended evidence regions with DDPM-corrupted features, then contrasts the three logit streams (original, positive, negative) at each decoding step with a confidence filter.

Key results

• On POPE (LLaVA-1.5-7B, random split), PND reaches 87.33 accuracy and 93.41 F1 versus 78.53/77.04 for regular decoding with the same backbone family (adversarial: 84.03 accuracy vs. 78.53 regular, 83.13 AGLA).
• On CHAIR with LLaVA-1.5-7B, PND lowers CHAIR_s from 51.0 to 46.0 and CHAIR_i from 17.6 to 14.0 while raising recall to 78.1.
• Ablation on POPE shows the full dual-path model (84.03 accuracy) outperforms positive-only (83.14) and negative-only (82.23) variants.

Surprising findings

• Reported deeper-layer attention to visual patches falls to about 4.9%, which the authors use to motivate the Bayesian imbalance framing.

Limitations

• Adds inference overhead from three forward passes and external BLIP-ITM attention extraction.
• Relies on a single external BLIP-ITM model for attention maps, so the proxy for visual evidence is not from the target VLM itself.
• Key hyperparameters (alpha, gamma, beta, lambda, tau, T) are sensitive, and several analyses are deferred to supplementary material.

Experimental setup

Datasets: MSCOCO A-OKVQA GQA

Benchmarks: POPE MME CHAIR GCCCE

Metrics: accuracy precision recall F1 CHAIR_s CHAIR_i total score

Baselines: VCD VAF AGLA

Builds on: Visual Contrastive Decoding DDPM BLIP-ITM cross-modal attention

Closest papers by meaning

• ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM (AAAI 2026 · similarity 0.92)
• Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Models (NeurIPS 2025 · similarity 0.92)
• Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding (ICML 2026 · similarity 0.91)
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.91)
• First Logit Boosting: Visual Grounding Method to Mitigate Object Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.91)
• SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.91)