Research map / Vision-language and multimodal / Multimodal LLMs

Locate-then-Sparsify: Attribution Guided Sparse Strategy for Visual Hallucination Mitigation

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LTS-FS, a plug-and-play framework that uses causal-intervention attribution over a bi-granularity hallucination dataset to locate hallucination-relevant LVLM layers and scales feature-steering intensity per layer (sparse mask plus soft weighting) to reduce hallucinations while preserving general ability.

Problem

Feature steering methods for hallucination mitigation in LVLMs apply uniform steering across all layers, which disrupts layers unrelated to hallucination and degrades general performance.

Approach

Attribute each layer's contribution to hallucinated token logits via attention-head masking interventions (token-level, and weighted-aggregated sentence-level), then map normalized scores to per-layer steering intensities with a threshold mask, so only hallucination-relevant layers are strongly steered.

Key results

• On CHAIR with LLaVA-v1.5-7B, LTS-FS(VTI) reaches CHAIR_S 35.8 vs 47.4 for VTI and 53.0 for regular decoding.
• On POPE Random with LLaVA-v1.5-13B, LTS-FS(VTI) raises accuracy from 84.08 (VTI) to 86.59.
• On LLaVA-Bench (GPT-4V judged), LTS-FS(Nullu) raises Qwen-VL2.5 detailedness from 5.68 (Nullu) to 6.07, and LLaVA-1.5 detailedness from 5.51 to 6.23.

Surprising findings

• Nullu achieves negligible gains on some MME perception tasks such as Count, suggesting uniform steering is brittle to feature distribution changes.
• Attribution calibration takes 1-2 hours on one A100, which is a non-trivial preparation cost compared with the inference-time gains.

Limitations

• Not evaluated on 70B-scale models, as existing feature steering methods have not been tested at that scale.
• Requires a preparation phase of about 1-2 hours of attribution per model, and relies on a small hand-annotated set of 200 samples.
• Sentence-level hallucination indicator requires factual-error labels, and soft gating results are comparable to hard gating, so the gain from the soft variant is limited.

Experimental setup

Datasets: CHAIR POPE Antidote MSCOCO GQA MME LLaVA-Bench

Benchmarks: CHAIR POPE MME LLaVA-Bench

Metrics: CHAIR_S CHAIR_I Recall Length Accuracy Precision F1 MME score GPT-4V accuracy GPT-4V detailedness

Baselines: Regular VCD AGLA Nullu VTI

Builds on: Nullu VTI VCD

Closest papers by meaning

• Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation (ICML 2026 · similarity 0.94)
• Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models (ICLR 2026 · similarity 0.93)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.92)
• Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.92)
• Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention (CVPR 2025 · similarity 0.92)