Research map / Vision-language and multimodal / Multimodal LLMs

Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TLVS, a token-level activation steering method for LVLM hallucination mitigation that extracts steering vectors from visually sensitive vs. insensitive tokens, refines them with RLHF-V corrections, and adapts steering strength per decoding step via a KL-based visual-sensitivity signal.

Problem

Hallucinations in large vision-language models; existing activation steering averages image-vs-no-image differences over all tokens with a fixed strength, diluting sparse visual signals and over-perturbing non-critical tokens.

Approach

Extract steering directions from teacher-forced visually sensitive vs. insensitive tokens (selected by |Δ_t|), refine them with a KL/proximal-regularized objective on RLHF-V corrections, and modulate steering strength per decoding step using a KL-based visual-sensitivity score with a confidence cap.

Key results

• On LLaVA-1.5-7B, TLVS raises POPE average from 85.09 to 89.78 and lowers AMBER Hal from 36.4 to 19.7.
• On LLaVA-1.5-7B, CHAIRs drops from 46.0 to 31.2 while Recall rises from 78.9 to 82.2.
• On Qwen2.5-VL-7B, TLVS lowers CHAIRs from 24.0 to 12.0 and AMBER Hal from 29.0 to 16.7, with POPE average 86.90 to 87.33.

Surprising findings

• Refinement alone on Qwen2.5-VL-7B reduces hallucinations but lowers coverage, which is recovered only by adaptive steering.
• On Qwen2.5-VL-7B, POPE gains are marginal (+0.43 average), while gains on HallusionBench and CHAIR are larger.

Limitations

• Roughly 2x inference overhead due to the auxiliary no-image route.
• Residual hallucinations remain due to bounded steering strength and imperfect step selection.
• Refinement uses human-corrected RLHF-V data, so it is not purely training-free.

Experimental setup

Datasets: RLHF-V

Benchmarks: POPE HallusionBench CHAIR (COCO) AMBER MMHal-Bench

Metrics: accuracy F1 CHAIRs CHAIRi Recall Cover Hal Cog qAcc fAcc

Baselines: VCD SPARC ASD SHARP VTI VISTA DPO (RLHF-V)

Builds on: activation steering VCD RLHF-V contrastive activation addition

Closest papers by meaning

• Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models (ICLR 2026 · similarity 0.95)
• Locate-then-Sparsify: Attribution Guided Sparse Strategy for Visual Hallucination Mitigation (CVPR 2026 · similarity 0.94)
• One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination (CVPR 2026 · similarity 0.94)
• Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.93)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.93)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)