Research map / Vision-language and multimodal / Multimodal LLMs

Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AIR, a training-free decoding-time method that reinforces visual tokens in MLLM FFN layers via prototype-based token reduction and OT-guided selection of image patches, reducing object hallucination on CHAIR and POPE.

Problem

Object hallucination in multimodal LLMs, where generated content contradicts visual evidence; existing visual re-injection methods inject all visual tokens indiscriminately, pulling attention to background regions.

Approach

Condense visual tokens via distance-to-prototype ranking, then select image patches whose embeddings are best aligned with hidden states using entropic optimal transport (Sinkhorn) distance, and re-inject only the selected patches into the FFN during decoding.

Key results

• On LLaVA-1.5-7B CHAIR, AIR reduces CHAIR_S from 22.0 to 18.4 and CHAIR_I from 6.7 to 5.7 with BLEU 14.4 vs 14.5 vanilla.
• On POPE MSCOCO random setting with LLaVA-1.5-7B, AIR reaches 89.0 accuracy and 88.2 F1 versus 83.7/83.0 vanilla.
• AIR adds roughly 0.02s latency (2.07s vs 1.68s) and 0.2 GB GPU memory over LLaVA-1.5-7B on a single A100.

Surprising findings

• Random patch injection slightly worsens CHAIR relative to the vanilla LLaVA-1.5-7B baseline (CHAIR_S 22.3 vs 22.0).
• On MME and MMBench with GLM-4V-9B, AIR matches the vanilla baseline exactly, and VAF drops MMBench overall to 14.78 on LLaVA-1.5 in the reported table.

Limitations

• Not explored for reasoning multimodal models or agents, per the authors.
• Efficiency overhead is reported only for CHAIR on one GPU; extra latency relative to vanilla is nonzero.
• Results on some benchmarks (e.g., MMBench for VAF, GLM-4V-9B) show identical or inconsistent numbers across tables, limiting clarity of gains.

Experimental setup

Datasets: MSCOCO val2014

Benchmarks: CHAIR POPE MME MMBench LLaVA-Bench HallusionBench V* Bench MMHal-Bench MM-Vet

Metrics: CHAIR_S CHAIR_I BLEU accuracy F1-score

Baselines: VCD MemVR VAF

Builds on: VCD MemVR Sinkhorn algorithm

Closest papers by meaning

• Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models (ICLR 2026 · similarity 0.93)
• One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination (CVPR 2026 · similarity 0.93)
• Learning to Decode Against Compositional Hallucination in Video Multimodal Large Language Models (ICML 2026 · similarity 0.93)
• ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large Language Models (CVPR 2025 · similarity 0.93)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.93)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)