Research map / Vision-language and multimodal / Multimodal LLMs

SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding

ICML 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SECOND, a training-free multi-stage decoding method that progressively selects object-relevant high-resolution patches via visual attention and contrasts outputs across stages to reduce object hallucination in VLMs.

Problem

Object hallucination in vision-language models, where generated answers are not grounded in the visual content; existing multi-scale VLM designs integrate visual information indiscriminately across scales.

Approach

Coarse-to-fine multi-stage patch selection driven by visual attention entropy, combined with multi-stage contrastive decoding that contrasts sparse early-stage outputs (amateurs) with dense final-stage outputs (expert), applied training-free to existing VLMs.

Key results

• On POPE, SECOND with CD outperformed baseline and VCD in 11 of 12 settings; e.g., LLaVA-NeXT Vicuna-7B MSCOCO f1 89.2 vs 86.5 baseline.
• On MMBench (lite) with LLaVA-NeXT Vicuna-7B, SECOND with CD reached 80.0 vs 75.8 baseline.
• Yi-VL with SECOND+CD improved MMStar from 34.8 to 39.8 and VQAv2 (lite) from 64.3 to 65.3.

Measured effects reported in the paper

Surprising findings

• Single-stage CD achieves the same POPE peak (89.2) as multi-stage CD, with multi-stage advantage mainly on MMStar.
• VCD on LLaVA-OneVision Qwen2-0.5B drops VQAv2 (lite) from 74.6 to 55.0.

Limitations

• Performance depends on internal attention maps, which can fail when attention is poorly calibrated.
• Multi-stage decoding increases decoding time relative to single-pass inference.
• Mistral-7B and some general-task results show SECOND+CD not uniformly better (e.g., MMBench for Mistral-7B and Qwen2-0.5B).

Experimental setup

Datasets: MSCOCO A-OKVQA GQA

Benchmarks: POPE VQAv2 (lite) MMStar MMBench (lite)

Metrics: recall accuracy f1 exact match accuracy

Baselines: baseline VLM (no decoding change) VCD LLaVA-NeXT

Builds on: Contrastive Decoding VCD LLaVA-NeXT POPE

Closest papers by meaning

• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.96)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.94)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.94)
• Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.94)
• ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models (AAAI 2025 · similarity 0.94)
• ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large Language Models (CVPR 2025 · similarity 0.93)