Research map / Vision-language and multimodal / Multimodal LLMs

Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization

NeurIPS 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Hallucination-Induced Optimization (HIO), which fine-tunes an "evil" LVLM via a Contrary Bradley-Terry preference objective to amplify hallucinations, then contrasts its logits with the original model at decoding to reduce object hallucinations.

Problem

LVLMs hallucinate objects, and existing visual contrastive decoding uses global visual noise that cannot precisely induce hallucinatory tokens, limiting or even creating new hallucinations.

Approach

Reverse the Bradley-Terry preference model so the hallucinated output is preferred, train an amplified-hallucination ("evil") LVLM on paired hallucinated/correct annotations from RLHF-V (with multi-hallucination and logit-gap constraints), and contrast its logits with the original model at inference.

Key results

• On POPE MSCOCO random setting, HIO reaches 90.21% accuracy for LLaVA-1.5 vs 87.73% for VCD and 83.29% for regular decoding.
• On CHAIR (COCO), the full HIO model reduces CHAIR_S from 50.0 to 41.4 and CHAIR_I from 15.4 to 10.5 relative to the regular baseline.
• Ablation on CHAIR-COCO shows CBTM, AMTH and ACI progressively lower CHAIR_S from 33.4 to 11.2 and CHAIR_I from 9.07 to 2.02.

Measured effects reported in the paper

Surprising findings

• The paper argues that naive DPO-style hallucination training does not reliably induce hallucinations meeting the contrast condition, motivating the reversed BT objective and extra constraints.

Limitations

• Findings establish only a necessary, not sufficient, condition for contrast decoding success.
• Inducing hallucinations requires additional training, adding computational cost compared with training-free contrast decoding.

Experimental setup

Datasets: RLHF-V MSCOCO A-OKVQA GQA

Benchmarks: POPE CHAIR MME

Metrics: accuracy precision recall F1 score CHAIR_S CHAIR_I MME total score

Baselines: Regular decoding VCD ICD VDD OPERA

Builds on: Visual Contrastive Decoding Direct Preference Optimization Bradley-Terry model OPERA

Closest papers by meaning

• The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs? (NeurIPS 2025 · similarity 0.94)
• ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models (AAAI 2025 · similarity 0.93)
• Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.93)
• SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.92)
• Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.92)
• Mitigating Object Hallucination in Large Vision-Language Models via False Discovery Controlled Visual Data Splitting (NeurIPS 2026 · similarity 0.92)