AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

HALMES: Knowing When to Intervene in LVLM Hallucination Mitigation

Ruining Hu, Jiaqi Lu, Xiao Liu, Ying Shen, Yu Wang, Lin Zhang

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HALMES, a method that decides when to apply contrastive decoding for LVLM hallucination mitigation, instead of intervening at every decoding step.

Problem

LVLMs produce object hallucinations inconsistent with visual content; contrastive decoding mitigations intervene at every decoding step, which may disrupt otherwise correct generation.

Approach

Selective, step-level intervention: contrastive decoding is applied only at decoding steps where hallucination risk is judged to be present, rather than indiscriminately at all stages, as a training-free inference-time method.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add HALMES selective triggeringVCD · MSCOCO val2014 (InternVL3.5-8B)CHAIR_S: 44.00 -> 38.20Improved
add HALMES selective triggeringVCD · AMBER Hal rate (InternVL3.5-8B)Hal rate: 38.67 -> 28.00Improved
add HALMES selective triggeringSelf-Aug · AMBER Hal rate (Qwen3VL-8B)Hal rate: 35.33 -> 38.33Hurt
add HALMES selective triggeringM3ID · AMBER (Qwen3VL-8B)CHAIR/Hal: Hal 34.67 -> 37.33Hurt
add HALMES selective triggeringSID · MMHal-Bench (Qwen3VL-8B)Hal rate: 43.01 -> 48.35Hurt
add HALMES selective triggeringVCD · MMHal-Bench (Qwen3VL-8B)Hal rate: 51.06 -> 40.66Improved
add HALMES selective triggeringCD methods · BingoScore: 3.50 -> 3.49 (VCD, Qwen)No clear effect
add HALMES selective triggeringCD methods · MSCOCO coverageCoverage: AVISC 53.39 -> 52.03Mixed
trigger CD randomly at 70% instead of on detected segments (ablation)CD · COCO subsetCHAIR_I: 13.11 -> 14.60Hurt
trigger CD periodically instead of on detected segments (ablation)CD · COCO subsetCHAIR_I: 13.11 -> 14.67Hurt
use HALMES trigger instead of random/periodic at similar frequency (ablation)CD · COCO subsetCHAIR_I: 14.60 -> 12.29Improved
use VAR from full layers rather than a layer subset (ablation)detector · detection AUROC (Qwen3VL-8B)AUROC: 90.64 -> 91.31Improved

Experimental setup

Builds on: contrastive decoding

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy