Research map / Vision-language and multimodal / Multimodal LLMs
CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs
Yassine Ouali, Adrian Bulat, Brais Martinez, Georgios Tzimiropoulos
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CLIP-DPO fine-tuning | LLaVA-1.5 7B · AMBER | AMBER score: 83.5 -> 89.6 | Improved |
| add CLIP-DPO fine-tuning | MobileVLM-v2 1.7B · AMBER | AMBER score: 81.4 -> 85.3 | Improved |
| use CLIP-DPO instead of HA-DPO | LLaVA-1.5 7B · AMBER | AMBER score: 86.7 -> 89.6 | Improved |
| add CLIP-DPO fine-tuning | MobileVLM-v2 1.7B · AMBER generative | CHAIR: 3.8 -> 4.2 | Hurt |
| add CLIP-DPO fine-tuning | LLaVA-1.5 7B · Zero-shot classification (9 datasets avg) | top-1 accuracy: 40.0 -> 47.4 | Improved |
| add HA-DPO | LLaVA-1.5 7B · Zero-shot classification (9 datasets avg) | top-1 accuracy: 40.0 -> 39.7 | No clear effect |
| add CLIP-DPO fine-tuning | LLaVA-1.5 7B · LLaVA benchmark (avg of 6) | average score: 68.8 -> 67.9 | Hurt |
| add CLIP-DPO fine-tuning | MobileVLM-v2 7B · LLaVA benchmark (avg of 6) | average score: 72.1 -> 72.3 | No clear effect |
| use alternative DPO-style loss (IPO, KTO, SLIC, cDPO) instead of DPO beta=0.1 (ablation) | CLIP-DPO · AMBER (MobileVLM-v2 1.7B) | AMBER score: 85.3 -> 83.1 to 83.6 | No clear effect |
| use a stronger CLIP scorer (ViT-H/14 DFN instead of ViT-L/14) (ablation) | CLIP-DPO · AMBER (MobileVLM-v2 1.7B) | AMBER score: 79.8 -> 85.3 | Improved |
Closest papers by meaning
- Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key (CVPR 2025 · similarity 0.94)
- Stop Learning it all to Mitigate Visual Hallucination, Focus on the Hallucination Target. (CVPR 2025 · similarity 0.93)
- Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMs (ICML 2026 · similarity 0.92)
- Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations (ICLR 2025 · similarity 0.92)
- P$^2$-DPO:Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization (ICLR 2026 · similarity 0.92)
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
