Research map / Vision-language and multimodal / Multimodal LLMs

Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Self-Introspective Decoding (SID), a training-free contrastive decoding method that amplifies vision-and-text association hallucinations by preserving the least-attended vision tokens after early decoder layers, then subtracts them from the original logits.

Problem

Hallucination in large vision-language models; existing contrastive decoding methods (VCD, ICD) disturb whole raw inputs, which adds uncertainty noise and doubles inference cost.

Approach

Instead of holistic input disturbances, LVLMs introspectively score vision token importance from attention in early decoder layers; the least-important vision tokens (CT²S) are kept to induce context- and text-aware fine-grained hallucinations that are then subtracted from the original logits, avoiding a second full forward pass over disturbed inputs.

Key results

• On POPE with LLaVA-1.5 7B under greedy decoding (adversarial), SID reaches 83.24 accuracy versus 79.11 for greedy decoding and 80.21 for ICD.
• On CHAIR with LLaVA-1.5 (greedy), SID lowers CHAIR_S from 49.6 to 44.2 and CHAIR_I from 14.4 to 12.2.
• SID reduces CD inference time by up to about 30% relative to contrastive baselines while keeping a single extra forward pass in early layers.

Measured effects reported in the paper

Surprising findings

• Enhancing vision information (adding high-attention tokens) gives less benefit than contrastive subtraction of the amplified hallucinations, especially in the adversarial setting.
• Pruning all vision tokens (ablating vision input) or selecting tokens in late layers does not improve over standard decoding, suggesting early-layer selection matters.

Limitations

• Layer index and pruning ratio are set manually as hyperparameters.
• Still produces hallucinations in fine-grained details such as eye color and vehicle identification, limited by the vision encoder.
• Training-free and does not use auxiliary analysis networks, so it inherits LVLM weaknesses.

Experimental setup

Datasets: MSCOCO POPE benchmark images (MSCOCO, A-OKVQA, GQA) Visual Genome

Benchmarks: CHAIR POPE GPT-4 assisted hallucination benchmark (Visual Genome) MME MMBench MathVista MMVet LLaVA-Bench MMMU GPT4-V assisted evaluation

Metrics: CHAIR_S CHAIR_I accuracy F1 score sentence-level hallucination ratio n-gram fluency words per image sentences per image MME score MMBench accuracy

Baselines: Sampling Greedy DoLa VCD ICD OPERA

Builds on: Visual Contrastive Decoding (VCD) Instruction Contrastive Decoding (ICD) Contrastive Decoding DoLa OPERA

Closest papers by meaning

• Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.95)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.95)
• Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.94)
• Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding (CVPR 2025 · similarity 0.94)
• MoLE:Decoding by Mixture of Layer Experts Alleviates Hallucination in Large Vision-Language Models (AAAI 2025 · similarity 0.94)
• Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.93)