Research map / Vision-language and multimodal / Multimodal LLMs

VASparse: Towards Efficient Visual Hallucination Mitigation via Visual-Aware Token Sparsification

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VASparse, a training-free, plug-and-play decoding method that combines visual-aware token sparsification, embedding-based sparse visual contrastive decoding, and a sinking-attention penalty to reduce visual hallucinations in LVLMs while keeping decoding speed.

Problem

Visual hallucinations in large vision-language models; existing decoding-based mitigation methods (contrastive decoding, rollback, secondary decoding) substantially slow inference, while naive visual-agnostic token sparsification worsens hallucinations.

Approach

Select KV tokens during decoding by a score combining attention and a visual-saliency term (image-token attention), contrast logits from vision-aware vs. vision-agnostic sparsified inputs using embeddings fed directly to the LM head (avoiding a second decoder pass), and down-weight attention sinks on low-semantic text tokens by accumulated attention.

Key results

• On CHAIR with LLaVA-1.5 (max 64 tokens), VASparse reaches CHAIR_i 5.82 and CHAIR_s 18.51, the lowest among compared methods, at 27.73 TPS.
• The paper reports up to 12.9x decoding speed over HALC and 6.4x over OPERA while achieving better hallucination scores.
• On offline POPE (F1, max 64 tokens), VASparse achieves the best F1 in most settings across LLaVA-1.5, MiniGPT-4 and mPLUG-Owl2.

Measured effects reported in the paper

Surprising findings

• Vision-agnostic token sparsification (FastV, SparseVLM, vanilla Top-K) increases hallucination as sparsity grows.
• Attention in LVLMs is highly sparse: the top 1% of tokens recall over 98% of attention score in most layers.

Limitations

• Sparsification is applied only after a minimum number of generated tokens (L_s), and hyperparameters (lambda, alpha, beta, sparsity rate) are fixed by manual setting without broad sensitivity analysis.
• Evaluation relies on CHAIR, POPE, MME, and GPT-4-judged SHR; MME results show HALC and OPERA gaining little, and the method's gains on MME are limited to short binary outputs.
• Theoretical optimality result is a simple top-S selection argument over a per-token score; it does not account for the interaction between selected tokens.

Experimental setup

Datasets: MSCOCO

Benchmarks: CHAIR POPE MME GPT-4 assisted SHR benchmark LLaVA-Bench

Metrics: CHAIR_i CHAIR_s TPS F1-score accuracy SHR

Baselines: Greedy Beam Search DoLa OPERA VCD SID HALC Woodpecker LURE FastV SparseVLM

Builds on: visual contrastive decoding FastV SparseVLM HALC OPERA DoLa

Closest papers by meaning

• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.93)
• Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.92)
• Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation (ICML 2026 · similarity 0.92)
• The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models Via Visual Information Steering (ICML 2025 · similarity 0.92)
• One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination (CVPR 2026 · similarity 0.92)
• Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models (ICLR 2025 · similarity 0.92)