Research map / Vision-language and multimodal / Multimodal LLMs

Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 14 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Rebuttal document for the AGLA paper (CVPR 2025) defending a training-free method that masks prompt-irrelevant image regions (via GradCAM and an image-text matching model) to mitigate object hallucinations in LVLMs, with new POPE, MMMU, OCRBench, and MMStar results.

Problem

Large vision-language models generate textual responses that mention objects not present in the image (object hallucination), partly because attention concentrates on prompt-irrelevant or prominent image features.

Approach

Use an image-text matching model with GradCAM to locate prompt-relevant regions, then mask out irrelevant image areas to build an augmented view whose contrast with the original view guides decoding, without modifying LVLM weights; the masking ratio adapts to matching confidence.

Key results

• On POPE, AGLA reaches F1 84.58 with 0.69s inference per sample, versus 83.55 for OPERA (1.64s) and 83.16 for VCD (0.56s).
• AGLA reduces false positives on POPE for LLaVA from 260 (regular) to 183, a 26.7% reduction, and false negatives from 320 to 269.
• On MMMU, OCRBench, and MMStar, AGLA improves LLaVA's overall scores (e.g., MMStar 0.31 to 0.34; OCRBench 250 to 299) and QWen-VL's overall scores.

Measured effects reported in the paper

Surprising findings

• Data augmentations (brightness, crop, flip, noise) degrade POPE F1 relative to AGLA, suggesting prompt-guided masking matters more than generic augmentation.

Limitations

• GradCAM localization is coarse and not precise for query objects, relying on the adaptive masking ratio to compensate.
• Integration into training is untested; the authors note it would require GPU resources not available for the rebuttal.
• Reported comparisons mostly cover POPE and CHAIR; the rebuttal's fine-grained and general benchmark results are limited to a few models.

Experimental setup

Datasets: POPE MMMU OCRBench MMStar CHAIR

Benchmarks: POPE CHAIR MMMU OCRBench MMStar

Metrics: F1 score false positives false negatives inference time accuracy CHAIR hallucination rate

Baselines: Regular decoding DOLA VCD OPERA

Builds on: visual contrastive decoding OPERA DoLa

Closest papers by meaning

• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.94)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
• Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations (ICLR 2025 · similarity 0.93)
• PAS: Prelim Attention Score for Detecting Object Hallucinations in Large Vision-Language Models (CVPR 2026 · similarity 0.93)
• HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models (ICML 2026 · similarity 0.93)
• SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.93)