Research map / Vision-language and multimodal / Multimodal LLMs

HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that attention-based object hallucination signals are confounded by token position and repetition (Simpson's paradox), and proposes HaloProbe, a Bayesian factorized detector whose token-level scores drive non-invasive beam search and post-hoc editing to reduce hallucinations in LVLM captions.

Problem

Attention-based signals for detecting object hallucinations in LVLM captions are unreliable because token position and object repetition confound them, and intervention-based mitigation methods degrade fluency.

Approach

A factorized Bayesian posterior: a class-balanced MLP over fine-grained internal features (layer/head attention, logit stats) gives a likelihood ratio, which is combined with a prior over external features (token position, repetition) estimated on the natural imbalanced distribution; the resulting probability scores candidates in beam search or guides post-hoc removal without modifying model inter

Key results

• HaloProbe detection on LLaVA-1.5-7B reaches 90.0% accuracy and 93.5 AUROC, versus 84.46 accuracy and 90.19 AUROC for DIML.
• With greedy decoding plus post-processing, LLaVA-1.5 CHAIR_s drops from 51.6 to 17.6 (HaloProbe PP) versus 25.0 for DIML, with F1 75.2 vs 76.1.
• On Qwen3-VL nucleus decoding, CHAIR_s falls from 25.2 to 14.4 with HaloProbe post-processing while F1 stays roughly flat (74.4 to 74.5).

Surprising findings

• Conditioning on token position reverses the attention trend between correct and hallucinated objects (Simpson's paradox), so globally averaged image attention can mislead.
• Attention intervention under greedy decoding increased repetition and loop-style degeneration (RE-2 up 64%, longest repeated span more than doubled).

Limitations

• Post-processing relies on an external LLM (GPT-5), adding cost and dependence on a closed model.
• Beam search cost grows linearly with beam size and only applies under stochastic decoding with temperature > 0.
• Many baseline numbers are copied from prior papers rather than re-run, and some cells are unavailable, limiting direct comparability.

Experimental setup

Datasets: MS COCO 2014 val

Benchmarks: MS COCO 2014 val (CHAIR) AMBER POPE

Metrics: accuracy AUROC precision recall F1 CHAIR_s CHAIR_i

Baselines: IC UT EAZY DIML PAI OPERA AD-HH AllPath Baseline decoding

Builds on: CHAIR PAI OPERA EAZY DIML

Closest papers by meaning

• Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations (ICLR 2025 · similarity 0.94)
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.93)
• Mitigating Object Hallucination in Large Vision-Language Models via False Discovery Controlled Visual Data Splitting (NeurIPS 2026 · similarity 0.93)
• Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention (CVPR 2025 · similarity 0.93)
• Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.93)
• Mitigating Hallucinations in Large Vision-Language Models by Adaptively Constraining Information Flow (AAAI 2025 · similarity 0.93)