Research map / Vision-language and multimodal / Multimodal LLMs

KVSmooth: Mitigating Hallucination in Multi-modal Large Language Models through Key-Value Smoothing

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes KVSmooth, a training-free plug-in that applies attention-entropy-adaptive EMA smoothing to KV-cache keys and values to reduce object hallucination in MLLMs.

Problem

Object hallucination in multi-modal LLMs grows as decoding proceeds (semantic drift), with visual grounding decaying over long generations.

Approach

Apply token-wise exponential moving average smoothing to KV-cache keys and values, with the smoothing coefficient set per token from the percentile rank of its attention row-entropy (a sink-degree proxy) within a FIFO queue, clipped around a reference value.

Key results

• On CHAIR with LLaVA-1.5, CHAIR_S drops from 41.8 to 18.2 while F1 rises from 77.5 to 79.2.
• On MiniGPT-4 CHAIR_S falls from 31.8 to 17.0 and on InstructBLIP from 61.4 to 42.2, with F1 gains to 71.7 and 75.1.
• On Object HalBench, CHAIR_SR is reduced by 63.1% for LLaVA-1.5 relative to baseline.

Surprising findings

• Applying EMA directly to hidden states (attention output) causes a severe recall drop, while smoothing keys and values jointly works best.

Limitations

• Gains on InstructBLIP are smaller in some metrics and CHAIR_S trades off with coverage on AMBER for LLaVA-1.5 and MiniGPT-4 (Cover drops for MiniGPT-4).
• Hyperparameters (lambda_ref, queue length, layer range) are tuned per model and benchmark, and results come from a 500-image CHAIR subset.

Experimental setup

Datasets: COCO 2014 validation (500 sampled images) AMBER OPOPE Object HalBench

Benchmarks: CHAIR OPOPE AMBER Object HalBench

Metrics: CHAIR_S F1 Accuracy Precision F-beta (0.2) Cover Hal Cog CHAIR_I CHAIR_SR

Baselines: VCD OPERA PAI SPARC MiddleLayer

Builds on: StreamingLLM OPERA Visual Contrastive Decoding PAI SPARC

Closest papers by meaning

• The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models Via Visual Information Steering (ICML 2025 · similarity 0.91)
• MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation (ICLR 2025 · similarity 0.91)
• Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation (ICLR 2026 · similarity 0.91)
• Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Models (NeurIPS 2025 · similarity 0.91)
• OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation (CVPR 2024 · similarity 0.91)
• MoLE:Decoding by Mixture of Layer Experts Alleviates Hallucination in Large Vision-Language Models (AAAI 2025 · similarity 0.91)