Research map / Vision-language and multimodal / Multimodal LLMs

SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Training-free SHIELD mitigates LVLM object hallucination by treating visual-encoder biases and vulnerability via caption-based token re-weighting, noise-token subtraction, and adversarially contrastive decoding.

Problem

Object hallucination in large vision-language models, which prior work has attributed to the LLM component; this paper traces it to the CLIP-style visual encoder through statistical bias, inherent bias, and adversarial vulnerability.

Approach

Three training-free modules on the visual encoder side: re-weight visual tokens by similarity to naive caption tokens (statistical bias), subtract the average encoder output on K noise inputs (inherent bias), and contrast logits from the natural image against those from an adversarially perturbed image (vulnerability).

Key results

• On CHAIR with LLaVA-1.5, SHIELD reduces C_S from 48.8 (vanilla) to 36.6 and C_I from 14.2 to 10.3, outperforming OPERA (44.6 / 12.8).
• On POPE COCO average, SHIELD reaches 87.0 accuracy with LLaVA-1.5 versus 81.3 vanilla and 84.7 OPERA.
• On MME hallucination subsets with LLaVA-1.5, SHIELD total score is 668.3 versus 565.3 vanilla; AMBER score 88.0 vs 82.0 vanilla.

Surprising findings

• Noise-image inputs trigger strongly skewed hallucinations toward specific classes (e.g., 'person' at 100%), while noise visual tokens do not, supporting the inherent-bias analysis.
• Adaptive plausibility constraint alone raised C_S slightly (48.8 to 50.2), showing filtering low-probability tokens does not suppress hallucination.

Limitations

• Decoding overhead is substantial: about 7.34 s per sample versus 2.59 s for vanilla, due to caption generation and adversarial optimization.
• Gains on InstructBLIP are limited because its Q-Former constrains use of modified visual features.
• Contrastive decoding reduces reliance on LLM prior knowledge, which hurts knowledge-dependent MME tasks such as celebrity, landmark, and artwork.

Experimental setup

Datasets: COCO 2014 val POPE LLaVA-Bench AMBER MME A-OKVQA GQA MMHal-Bench

Benchmarks: CHAIR POPE MME LLaVA-Bench AMBER MMHal-Bench

Metrics: CHAIR C_S CHAIR C_I accuracy F1 GPT-4o correctness and detailedness scores AMBER score MME total score

Baselines: VCD OPERA Vanilla LVLM DoLa ICD TAME SID LLaVA-RLHF CCA-LLaVA Less is More MCA-LLaVA FGSM

Builds on: Visual Contrastive Decoding OPERA CLIP adaptive plausibility constraint

Closest papers by meaning

• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.92)
• Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs (NeurIPS 2025 · similarity 0.92)
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.92)
• Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.92)
• On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models (NeurIPS 2025 · similarity 0.92)
• Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention (CVPR 2025 · similarity 0.91)