Research map / Vision-language and multimodal / Multimodal LLMs
Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Decoupling Contrastive Decoding (DCD), which trains separate positive and negative image projectors in LLaVA-1.5 on hallucination preference data and uses the learned negative projection as the contrastive image input at inference, avoiding DPO's likelihood displacement and VCD's handcrafted noise.
Problem
Hallucination in MLLMs: DPO-based alignment can cause likelihood displacement and degrade general reasoning, while training-free visual contrastive decoding relies on handcrafted image perturbations that may not reflect real hallucination patterns.
Approach
Decouple positive and negative preference samples into independently trained image projectors (positive and negative), avoiding pairwise optimization; at inference, contrast the original image's logits against logits from the learned negative projection instead of a noise-distorted image.
Key results
- On LLaVA-1.5-7B fine-tuned on BPO, DCD (Pos. & Neg.) reaches SEED 58.61 and POPE adversarial accuracy 83.73 vs. DPO 82.67, while DPO drops SEED by about 4 points (54.48 vs 58.57 base).
- Negative-only training on BPO raises HallusionBench to 40.17 vs. 39.22 for LLaVA-1.5 and POPE adversarial accuracy to 86.25 (Table 3 ablation).
- On Qwen 2.5 VL 3B, DCD (Pos. & Neg.) fine-tuned on RLHF-V gives average 55.85 vs. 54.75 base and MM-Vet 54.5 vs. 52.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train only on negative samples instead of DPO preference optimization | LLaVA-1.5-7B · BPO fine-tuning | Average: 36.21 -> 37.34 | Helped |
| Use decoupled positive and negative projection learning instead of DPO | LLaVA-1.5-7B · BPO fine-tuning | SEED: 54.48 -> 58.61 | Helped |
| Use decoupled positive and negative projection learning instead of DPO | LLaVA-1.5-7B · BPO fine-tuning | Hallusion: 37.85 -> 39.54 | Helped |
| Use decoupled positive and negative projection learning instead of DPO | LLaVA-1.5-7B · BPO fine-tuning | MMHal hallucination rate: 0.64 -> 0.60 | Helped |
| Use decoupled positive and negative projection learning instead of DPO | LLaVA-1.5-7B · BPO fine-tuning | POPE adversarial accuracy: 82.67 -> 83.73 | Helped |
| Use a learned negative projection instead of added image noise (ablation) | DCD · BPO; negative image embedding ablation | MM-Vet: 24.4 -> 29.4 | Helped |
| Use a learned negative projection instead of a randomly sampled other image (ablation) | DCD · BPO; negative image embedding ablation | Hallusion: 37.01 -> 40.17 | Helped |
| Initialize the negative projection with SFT-stage weights rather than random weights (ablation) | DCD · BPO; negative projection initialization ablation | MM-Vet: 26.1 -> 29.4 | Helped |
| Train only on negative rather than only positive samples (ablation) | DCD · BPO; positive and negative learning ablation | MM-Vet: 24.3 -> 29.4 | Helped |
| Train only on negative rather than only positive samples (ablation) | DCD · BPO; positive and negative learning ablation | POPE adversarial F1: 84.18 -> 85.11 | Helped |
| Use decoupled positive and negative projection learning instead of DPO | Qwen 2.5 VL 3B · BPO fine-tuning | MMHal hallucination rate: 0.69 -> 0.52 | Helped |
Surprising findings
- Training only on negative (hallucinated) responses gives most of the hallucination gains, while positive-only training yields marginal improvement.
- Randomly sampled other images as negatives partially preserve general ability but hurt HallusionBench, and noise-based VCD degrades SEED and MathVista on LLaVA-1.5.
Limitations
- Contrastive decoding doubles inference computation due to parallel processing of original and negative-projected features.
- Negative projection quality depends on hallucination patterns present in preference datasets, which mainly cover common object hallucinations and may underrepresent spatial or causal hallucinations.
- Evaluation largely relies on GPT-4o-judged benchmarks and results vary across backbones and datasets (e.g., Qwen 2.5 VL 3B results show mixed general-ability changes).
Experimental setup
Datasets: RLHF-V BPO RLAIF-V VLFeedback
Benchmarks: POPE MM-Vet MMHal HallusionBench SEED-Bench MMStar MMMU MathVista
Metrics: accuracy F1 GPT-4o score hallucination rate
Baselines: DPO SimPO VCD BPO Woodpecker
Builds on: Visual Contrastive Decoding Direct Preference Optimization LLaVA-1.5 Contrastive Decoding
Closest papers by meaning
- ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models (AAAI 2025 · similarity 0.93)
- The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs? (NeurIPS 2025 · similarity 0.92)
- Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation (ICLR 2026 · similarity 0.92)
- ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM (AAAI 2026 · similarity 0.92)
- AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding (NeurIPS 2025 · similarity 0.92)
- Hallucination-aware Intermediate Representation Editing in Large Vision-Lanugage Models (ICLR 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.