Research map / Vision-language and multimodal / Multimodal LLMs

Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Ensemble Decoding (ED), a training-free decoding method that splits an image into sub-images and ensembles their logits weighted by attention, plus a FastED variant, to reduce object hallucination in LVLMs.

Problem

Object hallucination in large vision-language models, where generated text mentions nonexistent objects or misdescribes existing ones; prior fixes need extra data, training, or external modules.

Approach

Split the input image into N sub-images, compute per-step attention-guided weights over sub-images from the LVLM's own (layer- and head-averaged) attention, and ensemble sub-image logits with the full-image logits; add an ED adaptive plausibility constraint that uses the weighted sub-image probabilities, and FastED which uses only the highest-weight sub-image.

Key results

• On POPE with LLaVA-1.5 7B, ED improves F1 by about 6.57 points and accuracy by 6.47 points on average over regular decoding, and beats AGLA in all three settings (e.g., Random F1 89.68 vs 87.71).
• On CHAIR subset, ED reaches the highest recall of 82.5 versus 78.9 for AGLA, with CHAIR_S 43.0 tied with AGLA and CHAIR_I 14.0.
• FastED runs about 2.36x faster than ED (6.96 vs 16.42 s per image latency) while keeping CHAIR_I at 13.1, though recall drops to 74.0.

Measured effects reported in the paper

Surprising findings

• ED struggles with object counting on MME, which the authors attribute to image splitting fragmenting objects.

Limitations

• Applies only to LVLMs with patch-wise projectors that preserve spatial locality, not resampler-based models.
• Lacks theoretical justification for its effectiveness.
• Multiple forward passes over sub-images make ED substantially slower than regular decoding.

Experimental setup

Datasets: POPE MSCOCO A-OKVQA GQA MME LLaVA-Bench DomainNet

Benchmarks: POPE CHAIR MME LLaVA-Bench

Metrics: precision recall F1 score accuracy CHAIR_S CHAIR_I accuracy and detailedness (1-10 Likert) latency

Baselines: Regular decoding DoLa OPERA VCD AGLA

Builds on: adaptive plausibility constraint visual contrastive decoding OPERA AGLA LLaVA-1.5

Closest papers by meaning

• SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.91)
• HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models (ICML 2026 · similarity 0.91)
• When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding (NeurIPS 2025 · similarity 0.91)
• Multi-Modal Hallucination Control by Visual Information Grounding (CVPR 2024 · similarity 0.91)
• Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.91)
• Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination (CVPR 2026 · similarity 0.90)