Research map / Vision-language and multimodal / Multimodal LLMs
Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MARINE, a training-free, API-free inference-time method that guides LVLM decoding with object lists from open-source detectors (DETR, RAM++) via classifier-free-style logit combination to reduce object hallucination.
Problem
Large vision-language models hallucinate non-existent objects in images; existing fixes need costly fine-tuning, curated data, or proprietary LLM APIs for post-hoc correction.
Approach
Extract object-level information from image-grounded vision models (DETR and RAM++, intersected for consensus), convert it to a textual guidance prompt, and combine conditional and unconditional LVLM logits with a classifier-free guidance weight gamma at decoding time.
Key results
- Average CHAIR_S 8.4 and CHAIR_I 3.7 across five LVLMs versus 11.0 and 5.2 for greedy decoding, with recall 44.5 vs 40.3.
- On POPE adversarial setting, average accuracy 80.4 vs 73.2 for greedy decoding; yes-ratio moves toward 50% (51.1 vs 67.0).
- Latency increases decoding time by 1.98x on LLaVA-7B, the lowest overhead among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DETR and RAM++ image-grounded guidance | Five LVLMs · MSCOCO | CHAIR_S: 10.1 -> 8.4 | Helped |
| Add DETR and RAM++ image-grounded guidance | Five LVLMs · MSCOCO | CHAIR_I: 4.8 -> 3.7 | Helped |
| Add DETR and RAM++ image-grounded guidance | Five LVLMs · POPE adversarial | Accuracy: 73.2 -> 79.9 | Helped |
| Add DETR and RAM++ image-grounded guidance | LLaVA · LLaVA-QA90 | detailedness: 4.59 -> 4.59 | No clear effect |
| Use DETR and RAM++ together rather than DETR alone (ablation) | MARINE · MSCOCO | CHAIR_S: 27.6 -> 17.8 | Helped |
| Use DETR and RAM++ together rather than RAM++ alone (ablation) | MARINE · MSCOCO | CHAIR_S: 29.0 -> 17.8 | Helped |
| Use intersection rather than union to combine detected objects (ablation) | MARINE · MSCOCO | CHAIR_S: 30.4 -> 17.8 | Helped |
| Adjust guidance strength dynamically using grounding confidence (ablation) | MARINE on LLaVA · MSCOCO | CHAIR_S: 17.8 -> 14.8 | Helped |
| Adjust guidance strength dynamically using grounding confidence (ablation) | MARINE on mPLUG-Owl2 · MSCOCO | CHAIR_S: 4.2 -> 5.0 | Hurt |
| Adjust guidance strength dynamically using grounding confidence (ablation) | MARINE on LLaVA · POPE adversarial | Accuracy: 66.9 -> 71.97 | Helped |
| Add DETR and RAM++ image-grounded guidance | LLaVA · MSCOCO image captioning | CIDEr: 0.08 -> 0.21 | Helped |
| Add DETR and RAM++ image-grounded guidance | mPLUG-Owl2 · MSCOCO image captioning | CIDEr: 1.05 -> 1.03 | Hurt |
Surprising findings
- Simple engineered hallucination-avoidance prompts often increase CHAIR scores, e.g., LLaVA CHAIR_S rises from 26.6 to 27.2 (direct) and 37.4 (as guidance).
- Union aggregation of detector outputs performed worse than intersection on LLaVA (CHAIR_S 30.4 vs 17.8).
Limitations
- Performance depends on the quality of the guidance detectors; MARINE-Truth shows the gap to an oracle is meaningful on some models.
- Only object hallucination is addressed; other hallucination types and broader benchmarks are not studied.
- Relies on GPT-4V for the main generation-quality judgment and on a limited sample of images for some evaluations.
Experimental setup
Datasets: MSCOCO A-OKVQA GQA LLaVA-QA90
Benchmarks: CHAIR POPE LLaVA-QA90 MSCOCO image captioning
Metrics: CHAIR_S CHAIR_I Recall accuracy F1 yes ratio GPT-4V accuracy score GPT-4V detailedness score BLEU ROUGE-L
Baselines: Greedy LURE Woodpecker VCD OPERA
Builds on: classifier-free guidance VCD OPERA LURE Woodpecker
Closest papers by meaning
- Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (ICLR 2024 · similarity 0.93)
- Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.93)
- Mitigating Object Hallucination in Large Vision-Language Models via False Discovery Controlled Visual Data Splitting (NeurIPS 2026 · similarity 0.93)
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
- HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Models (CVPR 2026 · similarity 0.93)
- Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention (CVPR 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.