Research map / Vision-language and multimodal / Multimodal LLMs
Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Antidote, a synthetic-data-driven DPO post-training framework where factual priors are injected into prompts for self-correction, plus CP-Bench, a benchmark for counterfactual presupposition questions, to reduce LVLM hallucinations.
Problem
LVLMs accept false presuppositions in counterfactual presupposition questions (CPQs) and hallucinate, while existing mitigation methods mainly target object existence and captioning hallucinations.
Approach
Synthesize decoupled images (object present/absent) with a diffusion model and grounding-based verification, then have the baseline LVLM self-correct its answer when given factual hints; use original (rejected) vs. self-corrected (preferred) responses for DPO trained with LoRA, with the hints removed from the training prompt.
Key results
- On CP-Bench test, LLaVA-1.5-7B F1 rises from 5.7 to 78.4 with Antidote, and LLaVA-1.5-13B from 17.3 to 83.5.
- On POPE average F1, LLaVA-1.5-13B improves from 85.67 to 88.99 (+3.32).
- CHAIR_s for LLaVA-1.5-7B drops from 19.4 to 9.4 and SHR from 36.7 to 18.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply Antidote post-training | LLaVA-v1.5-7B · CP-Bench-Test | F1-score: 5.7 -> 78.4 | Helped |
| apply Antidote post-training | LLaVA-v1.5-13B · CP-Bench-Test | F1-score: 17.3 -> 83.5 | Helped |
| apply Antidote post-training | LLaVA-1.5-7B · POPE average | F1-score: 86.07 -> 87.89 (+1.82%) | Helped |
| apply Antidote post-training | LLaVA-1.5-7B · CHAIR | CHAIR_s: 15.1 to 9.4 | Helped |
| apply Antidote post-training | LLaVA-1.5-7B · LLaVA-Wild | score: 65.4 -> 64.0 | Hurt |
| replace preference optimization with SFT (ablation) | LLaVA-1.5-7B + Antidote · POPE | F1-score: 87.89 -> 85.14 | Hurt |
| replace preference optimization with SFT (ablation) | LLaVA-1.5-7B + Antidote · MMBench | score: 65.4 -> 59.6 | Hurt |
| use LoRA rank 32 and alpha 64 (ablation) | LLaVA-1.5-7B + Antidote · CP-Bench | F1-score: 82.9 -> 12.7 | Hurt |
| use LoRA rank 128 and alpha 256 (ablation) | LLaVA-1.5-7B + Antidote · POPE | F1-score: 87.89 -> 85.67 | Hurt |
| use LoRA rank 256 and alpha 512 (ablation) | LLaVA-1.5-7B + Antidote · Antidote training | model stability | Hurt |
Surprising findings
- Post-training only on CPQs made the baseline overly cautious, so true presupposition questions had to be added to the training set.
- Open-source models such as InternVL2 and Qwen2-VL, which do well on general benchmarks, perform poorly on CP-Bench, suggesting over-learning of instruction tuning.
Limitations
- Relies on DeepSeek-V2 and Stable-Diffusion-3 for synthesis, and GPT-4o as judge for CP-Bench evaluation.
- Only LLaVA-family models are post-trained; generalization to other LVLM architectures is not shown.
- Post-trained models become more cautious, with a slight drop on LLaVA-Wild.
Experimental setup
Datasets: CC3M Visual Genome POPE CHAIR SHR
Benchmarks: CP-Bench POPE CHAIR SHR Science-QA MMBench MMVet LLaVA-Wild
Metrics: F1-score accuracy precision recall CHAIR_s CHAIR_i SHR
Baselines: VCD VDD HACL Volcano HA-DPO SeVa OPERA SID SFT
Builds on: DPO HA-DPO LoRA POPE CHAIR
Closest papers by meaning
- Envision, Attend, Then Respond: Counterfactual Hallucination Mitigation in Large Vision-Language Models (CVPR 2026 · similarity 0.93)
- Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats (NeurIPS 2025 · similarity 0.92)
- THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models (CVPR 2024 · similarity 0.92)
- Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning (AAAI 2025 · similarity 0.91)
- The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs? (NeurIPS 2025 · similarity 0.91)
- Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.