Research map / Vision-language and multimodal / Multimodal LLMs
Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeGF, a training-free decoding method that generates an image from an LVLM's initial response with a text-to-image model and uses the JS divergence between original- and generated-image next-token distributions to choose between complementary and contrastive decoding, reducing hallucinations.
Problem
LVLMs generate hallucinated responses inconsistent with the visual input; existing training-free contrastive decoding methods target language-prior or distortion-based biases and may not generalize to visual recognition errors such as counting.
Approach
Use a text-to-image diffusion model to reverse the initial response into an auxiliary visual reference image, then at each token compare next-token distributions under the original and generated images via JS divergence: enhance (complementary decoding) when they agree, contrast (contrastive decoding) when they diverge.
Key results
- On POPE MS-COCO random setting, DeGF reaches 89.03% accuracy with LLaVA-1.5, versus 88.87% for RITUAL (second-best).
- On CHAIR (64 tokens, LLaVA-1.5) DeGF lowers CHAIR_S to 18.4 from 21.4 for the next-best M3ID, and on InstructBLIP to 24.0 from 24.5 for HALC.
- On MME-Hallucination total score, DeGF achieves 644.44 with LLaVA-1.5 (vs 616.25 RITUAL) and 662.22 with Qwen-VL (vs 655.55 RITUAL).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DeGF instead of regular decoding | LLaVA-1.5 · MMVP | accuracy: 22.67% to 27.33% | Helped |
| Use DeGF instead of the strongest listed baseline | LLaVA-1.5 · MME-Hallucination total score | total score: 644.44 { ($\pm9.18$)} vs. 616.25 { ($\pm20.38$)} | Helped |
| Use DeGF instead of the strongest listed baseline | LLaVA-1.5 · MMBench | accuracy: 65.5 vs. 64.6 | Helped |
| Use DeGF instead of the strongest listed baseline | LLaVA-1.5 · CHAIR, max 64 tokens | CHAIR_S: 18.4 vs. 21.4 | Helped |
| Use DeGF instead of regular decoding | LLaVA-1.5 · POPE MS-COCO adversarial | accuracy: 77.43 -> 81.63 | Helped |
| Set the distance threshold to 0 instead of the default 0.1 (ablation) | DeGF · POPE MS-COCO random | accuracy: 0.6% | Hurt |
| Set the distance threshold to 1 instead of the default 0.1 (ablation) | DeGF · POPE MS-COCO random | accuracy: 1.1% | Hurt |
| Use the default threshold γ=0.1 rather than γ=0 or γ=1 (ablation) | DeGF · three benchmarks | performance | Helped |
| Use different Stable Diffusion model variants for visual-reference generation (ablation) | DeGF with LLaVA-1.5 · evaluated benchmarks | performance | No clear effect |
| Use SD-XL-v1.0 instead of SD-v1.5 for visual-reference generation (ablation) | DeGF with LLaVA-1.5 · evaluated benchmarks | performance | Helped |
| Use SD-XL-v1.0 instead of SD-v1.5 for visual-reference generation (ablation) | DeGF · image generation | generation time: 3.8 s/image vs 11.3 s/image | Hurt |
| Add generative visual feedback and self-correcting decoding | LLaVA-1.5 on CHAIR · CHAIR, max 128 tokens | CHAIR_S: 55.0 -> 48.8 | Helped |
Surprising findings
- Hallucinatory tokens show a long-tailed JS divergence distribution between original and generated-image predictions, and CLIP similarity between original and generated images correlates negatively with CHAIR_I (rho=-0.63).
- Inference latency is about 4x regular decoding (13.89s vs 3.44s per instance), making the method slower than contrastive baselines like VCD.
Limitations
- Relies on a text-to-image diffusion model, adding computational overhead and latency unsuitable for real-time use.
- Generated images may not faithfully reflect the original image, so feedback quality depends on the initial response and generator.
- Evaluated only on three LVLMs of about 7B parameters and on hallucination-focused benchmarks.
Experimental setup
Datasets: MS-COCO POPE CHAIR MME-Hallucination MMBench MMVP LLaVA-Bench
Benchmarks: POPE CHAIR MME-Hallucination MMBench MMVP LLaVA-Bench
Metrics: accuracy precision recall F1 CHAIR_S CHAIR_I length MME score MMBench accuracy GPT-4V-aided accuracy and detailedness
Baselines: Regular decoding VCD M3ID RITUAL Woodpecker HALC DoLa OPERA
Builds on: contrastive decoding VCD M3ID RITUAL adaptive plausibility constraints
Closest papers by meaning
- Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models (ICLR 2025 · similarity 0.95)
- Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.95)
- Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling (NeurIPS 2025 · similarity 0.95)
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.94)
- ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models (AAAI 2025 · similarity 0.94)
- Multi-Modal Hallucination Control by Visual Information Grounding (CVPR 2024 · similarity 0.94)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.