Research map / Vision-language and multimodal / Multimodal LLMs

ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models

AAAI 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ConVis, a training-free contrastive decoding method that uses a text-to-image model (Hyper-SDXL) to reconstruct the image from the MLLM's own caption, then contrasts logits from the original and reconstructed images to suppress hallucinated tokens.

Problem

MLLMs generate responses that do not match the input image (hallucination), and existing mitigation either needs costly post-processing/fine-tuning or uses decoding tricks that don't exploit visualized hallucinations.

Approach

Visualize hallucinations by feeding the MLLM's own (possibly hallucinated) caption to a T2I model, then use the logit difference between original-image and reconstructed-image decoding as a visual contrastive signal that penalizes hallucinated tokens; uses n=4 nucleus-sampled captions and images for diversity.

Key results

• On CHAIR, ConVis achieves the lowest CHAIR_S for all three backbones (e.g., 18.4 on LLaVA-1.5 vs 19.5 for Beam Search and 23.7 for VCD, per the table).
• On POPE average F1 across backbones, ConVis reaches 78.6 vs 77.4 for OPERA and 74.7 for VCD, with gains modest on LLaVA-1.5 and mPLUG-Owl2.
• On LLaVA-Bench with LLaVA-1.5, ConVis scores 73.3 overall vs 70.0 for Beam Search and 66.2 for VCD.

Measured effects reported in the paper

Surprising findings

• HALC scores 50.6 on LLaVA-1.5 POPE Random/Popular/Adversarial subsets (near chance), and only 47.1 on LLaVA-Bench, indicating failures in that setting.
• Nucleus sampling for caption generation (for image reconstruction) outperforms greedy captions, suggesting caption diversity helps hallucination visualization.

Limitations

• Strong dependence on the quality of the T2I model; performance gains on POPE are modest because captions do not cover question-specific objects.
• Uses a fixed captioning prompt rather than question-adaptive prompts, limiting VQA effectiveness.
• Adds computational overhead from generating n=4 images per query with a diffusion model.

Experimental setup

Datasets: MSCOCO val2014

Benchmarks: CHAIR HallusionBench POPE MME LLaVA-Bench

Metrics: CHAIR_S CHAIR_I fAcc aAcc F1 MME perception and cognition scores LLaVA-Bench relative score

Baselines: Greedy Search Nucleus Sampling Beam Search VCD OPERA HALC

Builds on: visual contrastive decoding (VCD) OPERA HALC Hyper-SDXL adaptive plausibility constraint

Closest papers by meaning

• The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs? (NeurIPS 2025 · similarity 0.94)
• Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.94)
• CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models (NeurIPS 2024 · similarity 0.94)
• SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.94)
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.93)
• Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Models (NeurIPS 2025 · similarity 0.93)