Research map / Vision-language and multimodal / Multimodal LLMs

ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ICT, a training-free method that trains per-head SVM classifiers on trusted vs. blurred-image activations and shifts selected attention heads at the image and object levels during the forward pass to reduce object hallucination in LVLMs.

Problem

Object hallucination in large vision-language models, where LVLMs over-rely on language priors and miss fine-grained visual details; existing fixes need costly fine-tuning or slow contrastive decoding that removes useful language priors.

Approach

Identify attention heads that encode image-level vs. object-level visual information by training per-head binary classifiers on activations from clean vs. blurred (globally or object-region) image pairs, then add precomputed activation shift vectors to the top-K selected heads during the forward pass, adding no decoding overhead.

Key results

• ICT improves average F1 on POPE by 7.09% for LLaVA-v1.5 and 5.44% for Qwen-VL across nine subsets, versus 2.19% and 1.14% for OPERA.
• On MME hallucination subset, ICT raises LLaVA-v1.5 score by 80.51 points and Qwen-VL by 34.67 points.
• Shift vectors from LLaVA-v1.5 transferred to Qwen-VL improve GQA F1 by about 4.62% on average; ICT adds essentially no inference latency compared with VCD (2.1-2.2x) and OPERA (3.4-3.6x) slowdowns.

Measured effects reported in the paper

Surprising findings

• Shift vectors computed on LLaVA-v1.5 transfer to Qwen-VL, suggesting model-agnostic trust directions.
• Image-level heads concentrate in early layers while object-level heads concentrate in later layers.

Limitations

• Requires access to model weights, so cannot be applied to closed-source LVLMs.
• Only Gaussian blur is used as the image transformation for constructing contrast pairs.
• Can fail on questions needing text-side reasoning combined with fine visual attributes (case study error).

Experimental setup

Datasets: POPE (COCO random subset) MSCOCO

Benchmarks: POPE MME LLaVA-Bench

Metrics: accuracy F1 score MME score inference latency

Baselines: VCD OPERA

Builds on: Visual Contrastive Decoding (VCD) OPERA Grounding DINO Inference-time intervention (ITI)

Closest papers by meaning

• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
• Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.93)
• Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance (ICML 2025 · similarity 0.92)
• Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.92)
• One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination (CVPR 2026 · similarity 0.92)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.92)