Research map / Vision-language and multimodal / Multimodal LLMs

Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes OPA-DPO, which uses GPT-4V-revised responses aligned on-policy via LoRA-SFT before DPO with image-focus and anchor terms, reducing LVLM hallucinations with only 4.8k samples.

Problem

DPO-based hallucination mitigation for LVLMs underperforms because preference data (especially expert-revised or injected responses) is off-policy with respect to the reference policy, so the KL constraint blocks learning the preferred responses.

Approach

Use GPT-4V to minimally revise hallucinated sentences, then LoRA-SFT the model on ground-truth and revised responses to make them on-policy (creating a new reference policy), and then run a DPO variant with hallucination-weighted language-correction loss, image-focus loss on distorted images, and anchored preference terms.

Key results

• OPA-DPO on LLaVA-1.5-7B with 4.8k data reaches AMBER HalRate 11.6 and Object HalBench CHAIRs 13.00, versus RLAIF-V (16k) at 16.2 and 16.00.
• OPA-DPO on LLaVA-1.5-13B achieves AMBER CHAIR 2.4 and HalRate 12.8 with 4.8k data.
• Ablation: removing OPA raises AMBER HalRate for 7B from 11.6 to 22.6 and Object HalBench CHAIRs from 4.25-level to 7.64 (CHAIRi).

Measured effects reported in the paper

Surprising findings

• Without OPA, DPO barely changes the log-probability of off-policy revised responses, and the policy tends to repeat sentences and fail to emit EOS when image focus or anchor terms are removed.
• Stronger hallucination suppression comes with reduced coverage and a conservative answering style that omits details.

Limitations

• Relies on GPT-4V for revisions and severity scores, which may themselves hallucinate.
• Conservative behavior lowers coverage and POPE accuracy is unchanged while precision rises.
• Evaluation limited to LLaVA-1.5 7B and 13B models.

Experimental setup

Datasets: RLAIF-V dataset

Benchmarks: AMBER MMHal-Bench Object HalBench POPE LLaVA-Bench

Metrics: CHAIR Cover HalRate Cog Score CHAIRs CHAIRi Accuracy Precision

Baselines: RLHF-V HA-DPO HSA-DPO HALVA POVID RLAIF-V mDPO LLaVA-RLHF Silkie

Builds on: DPO RLHF-V mDPO RLAIF-V LLaVA-1.5

Closest papers by meaning

• P$^2$-DPO:Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization (ICLR 2026 · similarity 0.94)
• Stop Learning it all to Mitigate Visual Hallucination, Focus on the Hallucination Target. (CVPR 2025 · similarity 0.94)
• Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization (NeurIPS 2025 · similarity 0.94)
• Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback (AAAI 2025 · similarity 0.93)
• Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMs (ICML 2026 · similarity 0.93)
• CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs (ICLR 2025 · similarity 0.92)