Research map / Vision-language and multimodal / Multimodal LLMs
VCGD: Visual Clue Guided Decoding with Caption Model for Mitigating Hallucination in Multimodal Large Language Models
Guoqing Chen, Fu Zhang, Bingqian Liu, Chenglong Lu, Jingwei Cheng
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VCGD, a visual-clue-guided decoding method that uses a caption model to mitigate hallucinations in multimodal LLMs.
Problem
Hallucinations in multimodal large language models, where generated outputs contain erroneous or fabricated information; existing contrastive-decoding work relies on manually perturbed inputs as references.
Approach
Instead of manually perturbing visual or instruction inputs to induce hallucinations, the method uses a caption model to derive visual clues that guide decoding of the MLLM. Only the abstract was available, so the exact mechanism is not described here.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train caption model (Qwen2.5-VL-3B) with RA-GRPO (ablation) | VCGD on InternVL-26B · MMVP | average accuracy: 44.8 -> 51.4 | Improved |
| train caption model (Qwen2.5-VL-7B) with RA-GRPO (ablation) | VCGD on InternVL-26B · MMVP | average accuracy: 48.0 -> 55.9 | Improved |
| train caption model (Qwen2.5-VL-7B) with RA-GRPO (ablation) | VCGD on InternVL-26B · LLaVA-Bench QA90 | overall score: 86.6 -> 94.8 | Improved |
| train caption model (Qwen2.5-VL-7B) with RA-GRPO (ablation) | VCGD on InternVL-26B · MMHal-Bench | hallucination rate: 33.3 -> 31.2 | Improved |
| scale caption model from 3B to 7B (no training) (ablation) | VCGD on InternVL-26B · MMVP | average accuracy: 44.8 -> 48.0 | Improved |
| scale caption model from 7B to 32B (no training) (ablation) | VCGD on InternVL-26B · MMVP | average accuracy: 48.0 -> 48.2 | No clear effect |
| use Qwen2.5-VL-72B instead of 7B caption model (no training) (ablation) | VCGD on InternVL-26B · MMVP | average accuracy: 48.0 -> 52.6 | Improved |
| use RA-GRPO-trained 7B instead of OpenAI o3 as caption model (ablation) | VCGD on InternVL-26B · MMVP | average accuracy: 52.9 -> 55.9 | Improved |
| add image confidence constraint on alpha_t (ablation) | VCGD with injected hallucinated clues · hallucination-injected clues | benchmark performance | Improved |
| use all four reward terms in Reward Agent rather than subsets (ablation) | VCGD caption model training · MMVP, POPE, MMHal-Bench | average | Improved |
| replace CODE with VCGD decoding | LLaVA-1.5-13B · MMVP | average accuracy: 34.0 -> 37.2 | Improved |
| replace CODE with VCGD decoding | LLaVA-NeXT-34B · MMVP | average accuracy: 42.6 -> 47.5 | Improved |
Experimental setup
Builds on: contrastive decoding
Closest papers by meaning
- ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models (AAAI 2025 · similarity 0.93)
- Learning to Decode Against Compositional Hallucination in Video Multimodal Large Language Models (ICML 2026 · similarity 0.93)
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.92)
- Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation (ICLR 2026 · similarity 0.92)
- AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding (NeurIPS 2025 · similarity 0.92)
- CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models (NeurIPS 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
