Research map / Vision-language and multimodal / Multimodal LLMs

Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DMAS, a training-free method that steers LVLM attention heads at inference using a semantically indexed database of truthfulness steering vectors plus a visual-perception steering vector, reducing hallucinations on MME, POPE, CHAIR.

Problem

Hallucinations in large vision-language models (LVLMs) such as fabricating objects or misdescribing image content; existing static activation steering uses one fixed truthfulness vector regardless of input semantics.

Approach

Cluster data semantically, extract a truthfulness steering vector per cluster from attention activations of truthful vs hallucinated answers, store them in a key-value database keyed by cluster embeddings, retrieve the most similar vector per query, and add it along with a noise-contrast visual perception vector to the top-K attention heads.

Key results

• On LLaVA-v1.5 7B, MME total score rises from 565.33 to 659.99 (+94.66) and CHAIR_S drops from 51.0 to 30.8.
• On POPE MSCOCO, average accuracy with LLaVA-v1.5 improves by 5.43% and F1 by 7.14% over the regular model.
• Ablation on CHAIR/POPE shows removing both vectors reverts CHAIR_S to 51.0, while using both gives 30.8.

Surprising findings

• Reducing the steering-vector dataset to 25% of its size has minimal effect on MME, MMVP and AMBER performance.
• Choosing hard same-category negatives versus random cross-category negatives for visual steering yields no significant difference.

Limitations

• Hyperparameters alpha, beta, K selected by grid search on evaluation benchmarks, with risk of overfitting.
• Results on several benchmarks and models are reported with inconsistent or incomplete baseline sets (e.g., some POPE entries lack precision/recall).

Experimental setup

Datasets: AMBER SEED MME POPE CHAIR MSCOCO GQA ScienceQA ViQuAE MMVP IllusionVQA HallusionBench

Benchmarks: MME POPE CHAIR AMBER ScienceQA ViQuAE MMVP IllusionVQA HallusionBench MM-Vet

Metrics: MME total score accuracy precision recall F1 CHAIR_S CHAIR_I AMBER score

Baselines: VCD OPERA VAF DECO DAMO AGLA ICT VTI ICD IBD DeFG CICD

Builds on: inference-time intervention (ITI) VTI ICT visual contrastive decoding

Closest papers by meaning

• Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation (ICML 2026 · similarity 0.95)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.94)
• Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.93)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
• Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation (ICLR 2026 · similarity 0.93)
• CausalLens: Sensitivity-Guided Multi-Head Causal Intervention for Hallucination Mitigation in Large Vision-Language Models (CVPR 2026 · similarity 0.93)