Research map / Vision-language and multimodal / Multimodal LLMs

Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision-Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RUDDER, a training-free single-pass steering method that extracts a per-sample visual evidence direction (CARD) from prefill self-attention residual updates and injects it during decoding with a Beta-Bernoulli-inspired adaptive gate to reduce object hallucination in LVLMs.

Problem

LVLMs over-rely on language priors during autoregressive decoding as visual information from the prefix dilutes, causing object hallucination; existing mitigations (contrastive decoding, iterative refinement, steering) add forward passes and latency.

Approach

Aggregate prefill self-attention residual updates into a normalized per-sample visual evidence vector (CARD) and re-inject it into the residual stream during decoding, scaled per token by a softplus-parameterized Beta-style gate on cosine alignment, so no extra forward pass is needed.

Key results

• On CHAIR with greedy decoding, RUDDER-Beta reduces CHAIR_S and CHAIR_I relative to vanilla, e.g., LLaVA-1.5 48.6→39.5 CHAIR_S and 13.6→10.5 CHAIR_I; averages reported as 33.2% and 28.6% relative reductions across three models.
• On POPE, RUDDER-Beta improves LLaVA-1.5 greedy accuracy from 85.34 to 86.53 and F1 from 84.91 to 86.03.
• RUDDER keeps about 96.0% of vanilla throughput on average (LLaVA-1.5 56.7→54.9 tokens/s), whereas VISTA drops to about 58.1% of vanilla.

Surprising findings

• RUDDER-Add (fixed strength) surpasses RUDDER-Beta on InstructBLIP POPE, suggesting adaptive gating is not uniformly beneficial.
• Authors report that the CHAIR-selected hyperparameters transfer well to POPE, with the CHAIR-tuned setting close to the best POPE configuration.

Limitations

• Hyperparameters (alpha_max, k, layer) must be tuned per architecture, as acknowledged by the authors.
• Gains on POPE are modest (around 0.5-1.6 points) and RUDDER-Add beats RUDDER-Beta on InstructBLIP.
• Some CHAIR_S comparisons are not uniformly best; e.g., DeGF and VISTA are stronger on some settings, and the Qwen2.5-VL and 13B comparisons are limited to few baselines.

Experimental setup

Datasets: MSCOCO

Benchmarks: CHAIR POPE MME

Metrics: CHAIR_S CHAIR_I recall accuracy F1 MME score throughput (tokens/s)

Baselines: DoLa VCD PAI HALC AGLA DeGF VISTA

Builds on: VISTA VCD DoLa Evidential Deep Learning

Closest papers by meaning

• The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models Via Visual Information Steering (ICML 2025 · similarity 0.93)
• Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidance (CVPR 2026 · similarity 0.93)
• Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation (ICLR 2026 · similarity 0.92)
• Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation (ICML 2026 · similarity 0.92)
• Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.91)
• Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.91)