Research map / Vision-language and multimodal / Multimodal LLMs
Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds sentence-level hallucination annotations from GPT-4/GPT-4V, trains an InternVL-based detector, uses a detect-then-rewrite pipeline with LLaVA to build preference pairs, and applies severity-weighted DPO (HSA-DPO) to reduce LVLM hallucinations.
Problem
LVLMs hallucinate (objects, attributes, relations not matching the input), and existing detection and mitigation work is coarse-grained, treats all hallucinations equally, and relies on expensive human or proprietary-model annotation.
Approach
Use proprietary-model sentence-level feedback (type, reason, severity) to train an open detector, use the detector plus a rewriter to automatically build preference pairs, and weight the DPO loss by the response's average hallucination severity so severe hallucinations are penalized more.
Key results
- Detection model reaches 85.60 claim-level and 86.94 segment-level accuracy on MHaluBench image-to-text, above GPT-4V with UNIHD (81.91 / 84.60).
- On LLaVA-1.5, HSA-DPO lowers Object HalBench CHAIR_S from 46.3 to 5.3 and AMBER Hal from 36.4 to 13.4 (about 76.3% and 36.1% relative reductions).
- Ablation: replacing fine-grained feedback with coarse GPT-4V feedback raises CHAIR_S from 5.3 to 17.0; removing severity weighting (vanilla DPO) raises it to 6.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use fine-grained AI feedback for detection instead of GPT-4V with UNIHD | LVLM hallucination detection · MHaluBench Claim | Mac.F1: 81.63 -> 85.52 | Helped |
| use fine-grained AI feedback for detection instead of GPT-4V with UNIHD | LVLM hallucination detection · MHaluBench Segment | Mac.F1: 81.71 -> 85.23 | Helped |
| use the proposed detection model instead of GPT-4V 2shot | LVLM hallucination detection · MFHaluBench Binary | F1: 74.4 -> 88.2 | Helped |
| use the proposed detection model instead of GPT-4V 2shot | LVLM hallucination detection · MFHaluBench Multi | ACC: 40.6 -> 74.3 | Helped |
| train with HSA-DPO instead of the LLaVA-1.5 base model | LLaVA-1.5 · Object HalBench | CHAIR_S: 46.3 -> 5.3 | Helped |
| train with HSA-DPO instead of the LLaVA-1.5 base model | LLaVA-1.5 · AMBER | Hal: 36.4 -> 13.4 | Helped |
| add hallucination severity weighting to DPO (ablation) | HSA-DPO with LLaVA-1.5 · Object HalBench | CHAIR_S: 6.7 -> 5.3 | Helped |
| add hallucination severity weighting to DPO (ablation) | HSA-DPO with LLaVA-1.5 · AMBER | Hal: 15.5 -> 13.4 | Helped |
| use detect-then-rewrite with the hallucination detection model instead of rewriting directly (ablation) | HSA-DPO with LLaVA-1.5 · Object HalBench | CHAIR_S: 42.1 -> 5.3 | Helped |
| use detect-then-rewrite with the hallucination detection model instead of rewriting directly (ablation) | HSA-DPO with LLaVA-1.5 · AMBER | Hal: 32.4 -> 13.4 | Helped |
| use fine-grained feedback instead of coarse-grained GPT-4V feedback (ablation) | HSA-DPO with LLaVA-1.5 · Object HalBench | CHAIR_S: 17.0 -> 5.3 | Helped |
| use fine-grained feedback instead of coarse-grained GPT-4V feedback (ablation) | HSA-DPO with LLaVA-1.5 · AMBER | Hal: 27.3 -> 13.4 | Helped |
Surprising findings
- The detector trained on a small annotated set outperforms GPT-4V and Gemini on MHaluBench detection despite using an open model.
- Relationship and action hallucinations improve only limitedly after mitigation training.
Limitations
- Relies on proprietary GPT-4/GPT-4V for the initial fine-grained annotations, so cost and dependency are not fully removed.
- Limited improvement on relationship and action hallucinations due to their scarcity in the preference data.
- Hallucination severity scores are AI-assigned and rely on GPT-4V ratings.
Experimental setup
Datasets: Visual Genome Silkie RLHF-V instruction set MHaluBench AMBER Object HalBench MMHal-Bench POPE LLaVA Bench in the wild
Benchmarks: MHaluBench MFHaluBench Object HalBench AMBER MMHal-Bench POPE LLaVA Bench in the wild
Metrics: Accuracy Precision Recall Macro F1 F1 CHAIR_S CHAIR_I CHAIR Coverage Hal
Baselines: GPT-4V Gemini UNIHD LRV LLaVA-RLHF RLHF-V Silkie POVID InstructBLIP Qwen-VL-Chat LLaVA-1.5
Builds on: DPO RLHF-V Silkie UNIHD LoRA
Closest papers by meaning
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.94)
- Stop Learning it all to Mitigate Visual Hallucination, Focus on the Hallucination Target. (CVPR 2025 · similarity 0.94)
- Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key (CVPR 2025 · similarity 0.93)
- Scalable Token-Level Hallucination Detection in Large Language Models (NeurIPS 2026 · similarity 0.93)
- Detecting and Preventing Hallucinations in Large Vision Language Models (AAAI 2024 · similarity 0.93)
- HaDeMiF: Hallucination Detection and Mitigation in Large Language Models (ICLR 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.