Research map / Vision-language and multimodal / Multimodal LLMs
Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Diagnoses why hallucination mitigation fails on cognitive prompts and proposes VDGD, a training-free decoding method that grounds generation in a self-generated image description via KL-based token scoring, plus the VaLLu benchmark.
Problem
Hallucination mitigation methods for LVLMs mainly help visual recognition prompts (object description) but not reasoning-intensive cognitive prompts, which the paper attributes to a visual perception gap.
Approach
Generate a detailed image description with the LVLM itself, prepend it to the instruction, and during decoding rank plausible top-k tokens by KL divergence to the prompt/description tokens (lower divergence preferred), sampling via softmax over negative KL.
Key results
- VDGD achieves the top score on VaLLu across all eight LVLMs, e.g. LLaVA-1.5 improves from 2.03 (vanilla greedy) to 2.64 (+30%).
- On MathVista, VDGD gives LLaVA-1.5 2.19 vs 1.65 vanilla greedy (+33%), the largest reported gain.
- GPT-4 scores correlate with expert human scores at about 0.97 on average, versus about 0.92 for original benchmark metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply VDGD decoding | LLaVA-1.5 · MMMU-Val | GPT score: 1.44 -> 1.62 | Helped |
| Apply VDGD decoding | LLaVA-1.5 · MathVista | GPT score: 1.68 -> 2.19 | Helped |
| Apply VDGD decoding | LLaVA-1.5 · MMBench | GPT score: 2.92 -> 3.12 | Helped |
| Apply VDGD decoding | LLaVA-1.5 · MME | GPT score: 3.53 -> 3.70 | Helped |
| Apply VDGD decoding | LLaVA-1.5 · Oven | GPT score: 2.65 -> 2.84 | Helped |
| Apply VDGD decoding | LLaVA-1.5 · LLaVA-Bench | GPT score: 3.10 -> 3.47 | Helped |
| Apply VDGD decoding | LLaVA-1.5 · MM-Vet | GPT score: 2.50 -> 3.01 | Helped |
| Apply VDGD decoding | LLaVA-1.5 · VaLLu | GPT score: 2.03 -> 2.64 | Helped |
| Use GPT-4 captions instead of LVLM-generated descriptions (ablation) | VDGD on LLaVA-1.5 · VaLLu | GPT score: 2.64 -> 2.91_{\pm0.6} | Helped |
| Use the small captioning model instead of LVLM-generated descriptions (ablation) | VDGD on LLaVA-1.5 · VaLLu | GPT score: 2.64 -> 2.38_{\pm0.08} | Hurt |
| Generate image descriptions without VCD (ablation) | VDGD on LLaVA-1.5 · VaLLu | GPT score: 2.64 -> 2.43_{\pm0.15} | Hurt |
| Remove KLD-based sampling (ablation) | VDGD on LLaVA-1.5 · VaLLu | GPT score: 2.64 -> 2.30_{\pm0.10} | Hurt |
Surprising findings
- Existing hallucination mitigation methods that reduce object hallucinations on AMBER do not improve reasoning benchmarks such as MMMU and MathVista.
- Text-only rephrased prompts yield better factual answers than image-text prompts, suggesting LVLMs have the knowledge but fail to integrate it with visual input.
Limitations
- Errors in the generated image description propagate into the final response.
- Requires prompting the LVLM twice, adding inference cost.
Experimental setup
Datasets: AMBER MMMU MathVista MATH-Vision MMC MME HallusionBench SynthDoG LLaVA-Bench MM-Vet MMBench Oven
Benchmarks: MMMU MathVista MMBench MME Oven LLaVA-Bench MMVET VaLLu
Metrics: GPT-4 judged factuality score (1-5) expert human correlation
Baselines: Vanilla-greedy Vanilla-sampling VCD OPERA Woodpecker LRV LURE PAI HALC
Builds on: Visual Contrastive Decoding OPERA Woodpecker LURE HALC
Closest papers by meaning
- Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning (AAAI 2025 · similarity 0.93)
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.92)
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.92)
- Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.92)
- MoLE:Decoding by Mixture of Layer Experts Alleviates Hallucination in Large Vision-Language Models (AAAI 2025 · similarity 0.92)
- Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.