Research map / Vision-language and multimodal / Multimodal LLMs
HALMES: Knowing When to Intervene in LVLM Hallucination Mitigation
Ruining Hu, Jiaqi Lu, Xiao Liu, Ying Shen, Yu Wang, Lin Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HALMES, a method that decides when to apply contrastive decoding for LVLM hallucination mitigation, instead of intervening at every decoding step.
Problem
LVLMs produce object hallucinations inconsistent with visual content; contrastive decoding mitigations intervene at every decoding step, which may disrupt otherwise correct generation.
Approach
Selective, step-level intervention: contrastive decoding is applied only at decoding steps where hallucination risk is judged to be present, rather than indiscriminately at all stages, as a training-free inference-time method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add HALMES selective triggering | VCD · MSCOCO val2014 (InternVL3.5-8B) | CHAIR_S: 44.00 -> 38.20 | Improved |
| add HALMES selective triggering | VCD · AMBER Hal rate (InternVL3.5-8B) | Hal rate: 38.67 -> 28.00 | Improved |
| add HALMES selective triggering | Self-Aug · AMBER Hal rate (Qwen3VL-8B) | Hal rate: 35.33 -> 38.33 | Hurt |
| add HALMES selective triggering | M3ID · AMBER (Qwen3VL-8B) | CHAIR/Hal: Hal 34.67 -> 37.33 | Hurt |
| add HALMES selective triggering | SID · MMHal-Bench (Qwen3VL-8B) | Hal rate: 43.01 -> 48.35 | Hurt |
| add HALMES selective triggering | VCD · MMHal-Bench (Qwen3VL-8B) | Hal rate: 51.06 -> 40.66 | Improved |
| add HALMES selective triggering | CD methods · Bingo | Score: 3.50 -> 3.49 (VCD, Qwen) | No clear effect |
| add HALMES selective triggering | CD methods · MSCOCO coverage | Coverage: AVISC 53.39 -> 52.03 | Mixed |
| trigger CD randomly at 70% instead of on detected segments (ablation) | CD · COCO subset | CHAIR_I: 13.11 -> 14.60 | Hurt |
| trigger CD periodically instead of on detected segments (ablation) | CD · COCO subset | CHAIR_I: 13.11 -> 14.67 | Hurt |
| use HALMES trigger instead of random/periodic at similar frequency (ablation) | CD · COCO subset | CHAIR_I: 14.60 -> 12.29 | Improved |
| use VAR from full layers rather than a layer subset (ablation) | detector · detection AUROC (Qwen3VL-8B) | AUROC: 90.64 -> 91.31 | Improved |
Experimental setup
Builds on: contrastive decoding
Closest papers by meaning
- The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs? (NeurIPS 2025 · similarity 0.93)
- Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.93)
- Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.93)
- MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation (ICLR 2025 · similarity 0.93)
- CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models (NeurIPS 2024 · similarity 0.93)
- Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling (NeurIPS 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
