Research map / Vision-language and multimodal / Multimodal LLMs
Stop Learning it all to Mitigate Visual Hallucination, Focus on the Hallucination Target.
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TL-DPO, a DPO variant that restricts preference learning to hallucinated response chunks and masked hallucination-related image regions, reducing object hallucinations in MLLMs such as LLaVA-1.5.
Problem
MLLMs generate object information absent from images; conventional preference learning (DPO/RLHF) over whole responses learns irrelevant signals and fails to correct object-level hallucinations efficiently.
Approach
Restrict DPO to hallucinated target chunks (token spans) and add a target-conditional loss that masks the bounding box of the hallucination-inducing object, so the model must rely on that image region to prefer the correct answer.
Key results
- On LLaVA-1.5, TL-DPO reduces CHAIR_s from 66.8 to 20.1 and CHAIR_i from 12.7 to 5.2, while MMHal rises from 2.42 to 2.72.
- TL-DPO improves LLaVA-1.5 MMBench from 63.0 to 67.8 and LLaVA-Bench from 63.4 to 71.2.
- On InternVL-2.5(8B), TL-DPO lowers CHAIR_s from 18.4 to 7.6 and raises MMHal from 3.3 to 3.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply TL-DPO instead of the baseline | LLaVA-1.5 · CHAIR_s | CHAIR_s: 66.8 -> 20.1 | Helped |
| Apply TL-DPO instead of the baseline | LLaVA-1.5 · CHAIR_i | CHAIR_i: 12.7 -> 5.2 | Helped |
| Apply TL-DPO instead of the baseline | LLaVA-1.5 · POPE | POPE: 85.90 -> 86.95 | Helped |
| Apply TL-DPO instead of the baseline | LLaVA-1.5 · MMHal | MMHal: 2.42 -> 2.72 | Helped |
| Apply TL-DPO instead of the baseline | LLaVA-1.5 · SciQA-IMG | SciQA-IMG: 66.8 -> 70.3 | Helped |
| Apply TL-DPO instead of the baseline | LLaVA-1.5 · MM-Vet | MM-Vet: 30.5 -> 32.2 | Helped |
| Apply TL-DPO instead of the baseline | LLaVA-1.5 · MMBench | MMBench: 63.0 -> 67.8 | Helped |
| Apply TL-DPO instead of the baseline | LLaVA-1.5 · LLaVA-Bench | LLaVA-Bench: 63.4 -> 71.2 | Helped |
| Add target condition loss (ablation) | TL-DPO on LLaVA-1.5 · CHAIR_s | CHAIR_s: 66.8 -> 32.4 | Helped |
| Add target condition loss (ablation) | TL-DPO on LLaVA-1.5 · POPE | POPE: 85.90 -> 84.40 | Hurt |
| Add target generation loss (ablation) | TL-DPO on LLaVA-1.5 · CHAIR_s | CHAIR_s: 66.8 -> 14.6 | Helped |
| Add target generation loss (ablation) | TL-DPO on LLaVA-1.5 · POPE | POPE: 85.90 -> 89.64 | Helped |
Surprising findings
- Llama3 with TL-DPO has slightly higher CHAIR_s (7.1 vs 5.5) and drops LLaVA-Bench from 65.8 to 58.5.
- Target-condition loss alone raises POPE less than target-generation alone, and TL-DPO on Qwen-VL-Chat lowers POPE (87.07 to 85.28) and MM-Vet.
Limitations
- Some comprehensive benchmark metrics decline for certain models (e.g., Llama3 on MM-Vet, LLaVA-Bench; Qwen-VL-Chat on SciQA-IMG).
- Sample-efficiency proposition relies on informal VC-dimension argument without empirical verification.
- Training data depends on GPT-4o-generated corrections and Visual Genome annotations.
Experimental setup
Datasets: Visual Genome
Benchmarks: CHAIR POPE MMHal-Bench SciQA-IMG MM-Vet MMBench LLaVA-Bench
Metrics: CHAIR_s CHAIR_i POPE accuracy MMHal score accuracy LLaVA-Bench score
Baselines: VLfeedback Human-Preference RLHF-V POVID HA-DPO HALVA
Builds on: DPO RLHF mDPO RLHF-V POVID
Closest papers by meaning
- Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback (AAAI 2025 · similarity 0.94)
- Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key (CVPR 2025 · similarity 0.94)
- Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization (NeurIPS 2025 · similarity 0.93)
- Detecting and Preventing Hallucinations in Large Vision Language Models (AAAI 2024 · similarity 0.92)
- Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection (CVPR 2026 · similarity 0.92)
- Automated Multi-level Preference for MLLMs (NeurIPS 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.