Research map / Vision-language and multimodal / Multimodal LLMs

Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidance

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Residual Decoding (ResDec), a training-free decoding method that aggregates confidence-weighted logits from JSD-identified stable historical tokens and fuses them into current logits to reduce language-prior hallucinations in LVLMs.

Problem

Large vision-language models suffer from language-prior hallucinations, where generated tokens become grammatically coherent but visually ungrounded, and existing mitigation methods need training or add substantial inference cost.

Approach

Observes that correct answer signals appear in logits of preceding tokens; uses JSD over a sliding window to find a stable "semantic anchoring" region, aggregates top-k candidate logits from those historical steps with entropy-based confidence weights, and blends them with current logits (alpha=0.5) plus a plausibility truncation (beta=0.1), needing only a single forward pass.

Key results

• ResDec improves POPE average accuracy by 7.84% and F1 by 8.01% over regular decoding across LLaVA-1.5, InstructBLIP and Qwen2.5-VL.
• On CHAIR, ResDec reduces CHAIR_S by 16.57% and CHAIR_I by 26.44% on average relative to regular decoding.
• ResDec achieves 29.11 ms/token latency versus 28.54 for greedy decoding, with memory 14296 MB vs 14257 MB for greedy.

Surprising findings

• Including the pre-semantic clarity phase (PSAP) in the historical window degrades performance relative to using only the semantic anchoring and expressive divergence phases.
• Top-4 confident-step selection performs worse than the full ResDec window selection on POPE and MME.

Limitations

• Relies on heuristic hyperparameters (window W=8, alpha=0.5, beta=0.1, candidate pool size 64-512) selected empirically.
• Some generated captions under ResDec show incoherent or repetitive phrasing (e.g., 'a a large cake'), per the case studies.
• Theoretical analysis in appendix relies on assumptions (alignment of residual with visual grounding) that are not empirically verified in detail.

Experimental setup

Datasets: POPE CHAIR HallusionBench MME MMBench MMVP ScienceQA MMStar MM-Vet SEEDBench2 Plus LLaVA-Bench (In-the-Wild)

Benchmarks: POPE CHAIR HallusionBench MME MMBench MMVP ScienceQA MMStar MM-Vet SEEDBench2 Plus LLaVA-Bench

Metrics: accuracy F1 CHAIR_S CHAIR_I fACC qACC aACC latency throughput memory

Baselines: VCD ICD DoLa OPERA AGLA ONLY MemVR VISTA

Builds on: visual contrastive decoding DoLa OPERA in-context decoding

Closest papers by meaning

• Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.94)
• Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models (ICLR 2025 · similarity 0.93)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.93)
• Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.93)
• SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.93)