Research map / Vision-language and multimodal / Multimodal LLMs

Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimization

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes IC-VCO, which places original and contrastive images in a shared multi-image context so the DPO partition functions cancel, adds a gated distillation term (VCDist) to a single-image branch, and builds hard negatives via targeted image editing, improving hallucination benchmarks on two 7B LVLMs.

Problem

Visual preference DPO for mitigating multimodal hallucination contrasts images with different partition functions, yielding a theoretically inconsistent objective, and synthetic or retrieved negatives are coarse-grained, enabling shortcut learning.

Approach

Place the original and contrastive images in one multi-image context with anchor prompts ("respond based on the first/second image") so both preference responses share the same conditioning and partition function; distill the multi-image preference into the single-image branch with dual-gated VCDist; and generate localized edited hard negatives with Qwen-Image-Edit.

Key results

• With edited samples on LLaVA-NeXT-Interleave-Qwen-7B, IC-VCO reaches an overall score of 63.35 vs 62.11 for the strongest baseline (SymMPO).
• With edited samples on LLaVA-OneVision-Qwen2-7B, IC-VCO reaches 66.82 overall vs 66.34 for S-VCO.
• Edited negatives have mean CLIP image similarity 94.72 vs 72.88 for synthetic negatives, indicating more fine-grained hard negatives.

Surprising findings

• Relation-centric metrics (AMBER-Rel, CRPE-Rel) are mixed; IC-VCO does not always beat baselines there, attributed to skew toward attribute and existence edits.
• Removing the anchor loss causes the largest ablation drop (63.35 to 61.15), and dropping the dual-gating or stop-gradient changes the score only slightly.

Limitations

• Relation edits are rare (828 of 19,453 samples), limiting gains on relation metrics.
• Edited data depends on external expert VLM (QwenVL-Plus) and image editor, which can introduce bias or artifacts.
• Gains over strongest baselines are modest (about 0.3 to 1.2 overall points).

Experimental setup

Datasets: SymMPO dataset VQA v2 MSCOCO TextVQA TPO dataset

Benchmarks: HallusionBench AMBER CRPE R-Bench BLINK

Metrics: accuracy aAcc fAcc qAcc overall macro-average score

Baselines: DPO mDPO V-DPO S-VCO SymMPO

Builds on: DPO mDPO V-DPO S-VCO SymMPO OPA-DPO

Closest papers by meaning

• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.92)
• Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models (CVPR 2026 · similarity 0.91)
• Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Models (NeurIPS 2025 · similarity 0.91)
• Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMs (ICML 2026 · similarity 0.91)
• Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization (NeurIPS 2024 · similarity 0.91)
• ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models (AAAI 2025 · similarity 0.91)