Research map / Vision-language and multimodal / Multimodal LLMs

REVIS: Sparse Latent Steering to Mitigate Object Hallucination in Large Vision-Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes REVIS, a training-free latent steering method that orthogonally projects a visual vector away from the language-prior direction and injects it sparsely at one calibrated deep layer with a risk-gated trigger to reduce object hallucination in LVLMs.

Problem

Object hallucination in large vision-language models, attributed to visual features becoming entangled with language priors in deep layers; prior inference-time fixes either add latency (contrastive decoding) or entangle steering vectors that cause collapse and utility loss.

Approach

Decompose the raw visual steering vector (ground-truth minus no-image states) by removing its projection onto the language-prior vector (hallucinated minus refusal states), inject only this orthogonal visual vector at a single calibration-selected deep layer, and gate injection by a percentile-based hallucination-risk threshold at inference.

Key results

• On Qwen2.5-VL-7B, REVIS reduces CHAIR_S from 31.00% to 25.00% (about 19% relative) and raises MM-Vet from 70.18 to 72.16 while VTI drops MM-Vet to 56.38.
• On LLaVA-1.5, CHAIR_S falls from 54.00% to 30.00% and MME overall rises from 1754.33 to 1787.27.
• Time per token is about 0.025s, matching regular decoding, versus about 0.05s for VCD and M3ID.

Surprising findings

• Naively injecting the raw visual vector causes model collapse (e.g., infinite repetition, or outputs such as 'image.jpg') at high intensity, and CHAIR drops to zero only through degeneration.
• Removing the dynamic threshold causes model collapse on LLaVA-1.5 under static steering.

Limitations

• Relies on the base model's representations and cannot fix absolute perceptual blindness.
• Evaluated only up to 7B/8B and 32B models; scalability to 70B+ untested.
• Hyperparameters (alpha, k, layer) are model-specific and tuned per model via calibration.

Experimental setup

Datasets: Nullu COCO train 2014 COCO val2014 POPE CHAIR MME MM-Vet MMMU-Pro

Benchmarks: POPE CHAIR MME MM-Vet MMMU-Pro

Metrics: accuracy recall F1 CHAIR_S CHAIR_I MME perception/cognition MM-Vet score MMMU-Pro accuracy time per token

Baselines: Regular decoding VCD M3ID ONLY AGLA VTI

Builds on: Activation steering VTI Nullu POPE CHAIR

Closest papers by meaning

• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.91)
• Mitigating Hallucinations in Large Vision-Language Models by Adaptively Constraining Information Flow (AAAI 2025 · similarity 0.91)
• Mitigating Object Hallucination in Large Vision-Language Models via False Discovery Controlled Visual Data Splitting (NeurIPS 2026 · similarity 0.90)
• Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (ICLR 2024 · similarity 0.90)
• Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating (ICML 2026 · similarity 0.90)
• Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models (ICLR 2026 · similarity 0.90)