Research map / Vision-language and multimodal / Multimodal LLMs
Systematic Reward Gap Optimization for Mitigating VLM Hallucinations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Topic-level Preference Rewriting (TPR), which builds DPO preference pairs by swapping topic-level units with the VLM's own resampled alternatives, plus a curriculum that progressively hardens rejected responses, reducing hallucinations in LLaVA-1.5-7B.
Problem
Preference data for DPO-based hallucination mitigation in VLMs lacks systematic control over reward gaps within preference pairs; ranking and external-rewriting methods give weak or mismatched signals.
Approach
Decompose responses into semantic units, cluster them into topics by textual and CLIP-based visual consistency, self-resample alternatives per topic with the reference model, and build (y_w, y_l) by selectively replacing units with high- or low-scored alternatives. A curriculum shifts rejected responses from easy (lowest-scored) to harder (bottom 20-40%) alternatives.
Key results
- TPR-CL-7B reaches CHAIR_s 3.4 and CHAIR_i 1.8 on Object HalBench, versus 53.6/25.2 for LLaVA-1.5-7B (about 93% reduction).
- TPR-CL-7B reduces MMHal hallucination rate to 30.2% from 51.0% for the base model, with score 3.06 vs 2.36.
- TPR-CL-7B reaches RefoMB trustworthiness 61.1 and win rate 32.3, versus 30.8 and 12.1 for the base model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply TPR curriculum learning instead of the unaligned baseline | LLaVA-1.5-7B · Object-HalBench | CH_s: 53.6 -> 3.4 | Helped |
| Apply TPR curriculum learning instead of the unaligned baseline | LLaVA-1.5-7B · MMHal-Bench | Hall.: 51.0 -> 30.2 | Helped |
| Use curriculum learning rather than greedy replacement (ablation) | TPR on LLaVA-1.5-7B · Object-HalBench | CH_s: 4.0 -> 3.4 | Helped |
| Replace greedy alternative selection with random selection (ablation) | TPR · Object-HalBench | CH_s: 5.9 -> 29.7 | Hurt |
| Replace greedy alternative selection with random selection (ablation) | TPR · AMBER | Acc.: 82.1 -> 78.9 | Hurt |
| Apply selective replacement only to the preferred response (ablation) | TPR · Object-HalBench | CH_s: 5.9 -> 6.4 | Hurt |
| Remove in-context rewriting (ablation) | TPR · Object-HalBench | CH_s: 5.9 -> 35.5 | Hurt |
| Remove multiple initial candidate responses (ablation) | TPR · Object-HalBench | CH_s: 5.9 -> 7.2 | Hurt |
| Remove intra-topic self-resampling (ablation) | TPR · Object-HalBench | CH_s: 5.9 -> 12.6 | Hurt |
| Replace selective topic replacement with ranking (ablation) | preference curation for LLaVA-1.5-7B · Object-HalBench | CH_s: 5.9 -> 9.7 | Hurt |
| Replace topic rewriting with external rewriting of both responses (ablation) | preference curation for LLaVA-1.5-7B · Object-HalBench | CH_s: 5.9 -> 15.6 | Hurt |
| Progressively make rejected responses harder during curriculum training (ablation) | TPR on LLaVA-1.5-7B · RefoMB Quantities | score: +20.8 | Helped |
Surprising findings
- The paper reports that an external-rewriter (GPT-4V-based) approach introduces hallucination-type distributions that deviate from the model's own failure modes.
- Each step of the curriculum adds gains on hard categories, with TPR-CL improving Quantities and Spatial Relations by +20.8 and +35.7 points over the greedy variant on RefoMB.
Limitations
- Pipeline has many sequential stages (decomposition, clustering, resampling, rewriting), raising complexity and compute cost.
- No error bars or statistical significance tests are reported.
- Focused on perceptual hallucinations; not evaluated on reasoning hallucinations.
Experimental setup
Datasets: VQA v2 MSCOCO ShareGPT-4V TextVQA MovieNet OKVQA Google Landmark v2 RLHF-V prompts
Benchmarks: Object HalBench MMHal-Bench AMBER POPE RefoMB LLaVA-Bench MMStar
Metrics: CHAIR_s CHAIR_i MMHal score MMHal hallucination rate AMBER accuracy AMBER F1 POPE F1/accuracy RefoMB trustworthiness RefoMB win rate LLaVA-Bench accuracy
Baselines: LLaVA-RLHF RLHF-V Silkie POVID MFPO HA-DPO OPA-DPO mDPO AMP-MEG RLAIF-V FGAIF HSA-DPO
Builds on: DPO RLAIF-V RLHF-V Louvain method CLIP
Closest papers by meaning
- Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMs (ICML 2026 · similarity 0.92)
- Calibrated Self-Rewarding Vision Language Models (NeurIPS 2024 · similarity 0.91)
- Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key (CVPR 2025 · similarity 0.91)
- CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs (ICLR 2025 · similarity 0.91)
- IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucination (ICML 2026 · similarity 0.90)
- Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.