Research map / Vision-language and multimodal / Multimodal LLMs
Detecting and Preventing Hallucinations in Large Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces M-HalDetect, a 16k-annotation sub-sentence-level hallucination detection dataset for detailed image descriptions, and uses it to train reward models and a fine-grained DPO variant (FDPO) that reduce InstructBLIP hallucinations.
Problem
LVLMs such as InstructBLIP generate detailed image descriptions containing hallucinations (non-existent objects, unfaithful descriptions, inaccurate relationships), and existing hallucination detection resources mostly consider only object hallucination and are coarse-grained.
Approach
A sub-sentence-level human-annotated hallucination dataset (Accurate/Inaccurate/Analysis/Unsure labels) is used to train fine-grained multimodal reward models (initialized from InstructBLIP) for best-of-n rejection sampling, and a Fine-grained DPO loss that uses segment-level labels instead of preference pairs.
Key results
- FDPO reduces InstructBLIP hallucination rate by 41% and best-of-64 rejection sampling by 55% (human evaluation).
- Reward model with best-of-16 sampling reduces hallucinations in LLaVA by 15% and mPLUG-OWL by 57%.
- Sentence-level binary reward model reaches 79.2% accuracy and 78.37 F1 on the M-HalDetect dev split.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-grained DPO ignoring analysis class (FDPO IA) on InstructBLIP | InstructBLIP · M-HalDetect human eval | Human Eval (fraction truthful): 0.71 to 0.83 at T=0 | Helped |
| Marking analysis as dispreferred class in FDPO (DA) vs ignoring it (ablation) | InstructBLIP FDPO · M-HalDetect human eval | Human Eval (fraction truthful): IA 0.83 vs DA 0.70 at T=0; IA 0.75 vs DA 0.58 at T=1 | Hurt |
| Best-of-64 rejection sampling with ternary reward model | InstructBLIP · M-HalDetect human eval | Human Eval (fraction truthful): 0.71 to 0.87 | Helped |
| Worst-of-64 selection instead of best-of-64 (ablation) | InstructBLIP · M-HalDetect human eval | Human Eval (fraction truthful): 0.53 vs 0.71 baseline | Hurt |
| Best-of-16 rejection sampling on LLaVA with InstructBLIP-trained reward model | LLaVA · M-HalDetect human eval | Human Eval (fraction truthful): 0.805 to 0.834 | Helped |
| Best-of-16 rejection sampling on mPLUG-OWL with InstructBLIP-trained reward model | mPLUG-OWL · M-HalDetect human eval | Human Eval (fraction truthful): 0.476 to 0.707 | Helped |
| Ternary vs binary reward model for sentence-level classification (ablation) | Reward model · M-HalDetect dev classification | Accuracy: 79.2 binary vs 71.4 ternary (sentence level) | Hurt |
| Segment-level vs sentence-level reward prediction (binary) (ablation) | Reward model · M-HalDetect dev classification | Accuracy: 79.2 sentence to 83.92 segment | Helped |
Surprising findings
- Marking analysis as a negative class in FDPO does not help and may worsen hallucination rates at higher temperature.
- LLaVA's baseline performs strongly, which the authors attribute to its training for detailed descriptions; GPT-4 judging was found unreliable and was not used.
Limitations
- Only one cycle of feedback collection and training was performed; some overfitting observed (more generic descriptions).
- Rejection sampling is prohibitively slow for real-world inference.
- Segment-level reward model cannot be used for rejection sampling without a localization model.
Experimental setup
Datasets: COCO val2014 LLaVA-150k
Benchmarks: M-HalDetect
Metrics: accuracy F1 score human-evaluated truthfulness rate reward model score
Baselines: InstructBLIP baseline (T=0) LLaVA mPLUG-OWL GPT-4 evaluation
Builds on: DPO InstructBLIP POPE
Closest papers by meaning
- Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback (AAAI 2025 · similarity 0.93)
- Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.93)
- Seeing Is Believing: Rich-Context Hallucination Detection for MLLMs via Backward Visual Grounding (AAAI 2026 · similarity 0.93)
- Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.92)
- Stop Learning it all to Mitigate Visual Hallucination, Focus on the Hallucination Target. (CVPR 2025 · similarity 0.92)
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.