Research map / Vision-language and multimodal / Multimodal LLMs

Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Prefill-Time Intervention (PTI), a training-free, modality-aware steering of the initial KV cache applied once at prefill, to reduce object hallucinations in LVLMs.

Problem

Decoding-time steering vectors reduce hallucination frequency but amplify the severity of residual hallucinations through autoregressive error accumulation ("snowball hallucinations") in large vision-language models.

Approach

Intervene only once on the prefill KV cache, with separate visual and textual steering directions derived from object-vs-background contrasts: keys are shifted toward visually grounded objects and values are shifted to filter background noise.

Key results

• On LLaVA-1.5 with greedy decoding, PTI reduces CHAIR_S from 47.4 to 15.4 and CHAIR_I from 13.7 to 5.4, outperforming VISTA (20.4 / 6.9).
• On POPE adversarial split, PTI reaches 82.21 average accuracy on LLaVA-1.5 versus 79.88 for vanilla, and 85.69 on Qwen-VL-Chat versus 83.69.
• PTI adds negligible latency (about 1.00-1.02x vanilla) whereas VCD and PAI roughly double latency.

Surprising findings

• Visual-only key-plus-value intervention on all visual tokens achieves much of the CHAIR gain but lowers F1 on LLaVA-1.5 (72.7 vs 75.3 vanilla for the full PTI configuration), indicating a generation-quality trade-off.
• Existing decoding-time steering methods such as VTI and VISTA yield only marginal POPE gains and even reduce F1 on Qwen-VL-Chat.

Limitations

• Visual intervention can reduce F1 score, suggesting over-emphasis on visual details may degrade generation quality.
• Steering strengths require grid search, and simultaneously maximizing key and value strengths degrades F1 and shortens outputs.

Experimental setup

Datasets: MSCOCO CHAIR POPE AMBER MMHal-Bench MME

Benchmarks: CHAIR POPE AMBER MMHal-Bench MME

Metrics: CHAIR_S CHAIR_I accuracy F1-score average score hallucination rate latency throughput

Baselines: VCD OPERA PAI VTI VISTA

Builds on: visual contrastive decoding VISTA VTI PAI OPERA

Closest papers by meaning

• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.95)
• CausalLens: Sensitivity-Guided Multi-Head Causal Intervention for Hallucination Mitigation in Large Vision-Language Models (CVPR 2026 · similarity 0.94)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.94)
• Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.93)
• Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models (ICLR 2026 · similarity 0.93)
• Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models (ICLR 2025 · similarity 0.93)