Research map / Vision-language and multimodal / Multimodal LLMs
HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HALC, a plug-and-play decoding algorithm that reduces object hallucination in LVLMs via adaptive focal-contrast grounding over sampled field-of-view crops plus a visual-matching beam search (BLIP-scored), without extra training.
Problem
Object hallucination (existence, attribute, relationship) in large vision-language models during autoregressive generation, where text priors progressively dominate visual evidence.
Approach
At object-related tokens, sample nested field-of-view crops around a Grounding DINO/OWLv2 box, select the most divergent (JSD) FOV pairs, bidirectionally contrast their logits to produce candidate tokens, and rerank full beams with a global BLIP image-text matching score.
Key results
- On MSCOCO CHAIR with MiniGPT-4, HALC reduces CHAIR_S from 30.87 (greedy) to 17.80 and CHAIR_I from 12.33 to 8.10 while keeping BLEU near 14.9.
- With LLaVA-1.5, HALC achieves CHAIR_S 13.80 and CHAIR_I 5.50, versus 20.80 and 6.77 for greedy decoding.
- HALC reports roughly 2.4x greedy decoding cost without parallelization and about 1.35x with parallel FOV decoding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add HALC (adaptive focal-contrast decoding with visual-matching beam search) over greedy decoding | MiniGPT-4 · MSCOCO CHAIR | CHAIR_S: 30.87 -> 17.80 | Helped |
| Use HALC instead of greedy decoding | LLaVA-1.5 · MSCOCO CHAIR | CHAIR_S: 20.80 -> 13.80 | Helped |
| Use HALC instead of greedy decoding | mPLUG-Owl2 · MSCOCO CHAIR | CHAIR_S: 23.20 -> 17.33 | Helped |
| Use HALC instead of greedy decoding | LLaVA-1.5 and MiniGPT-4 and mPLUG-Owl2 · MSCOCO CHAIR | BLEU: roughly equal to greedy (e.g. 16.10 vs 15.93 on LLaVA-1.5; 14.91 vs 14.33 on MiniGPT-4) | No clear effect |
| Use HALC instead of OPERA/LURE/Woodpecker SOTA baselines | MiniGPT-4 CHAIR_S · MSCOCO CHAIR | CHAIR_S: best baseline LURE 27.88 -> HALC 17.80 | Helped |
| Use detector (Grounding DINO) output as FOV sampling initialization rather than the full original image (ablation) | HALC · MSCOCO CHAIR | CHAIR_S: 27.8 -> 22.0 | Helped |
| Use Grounding DINO detector initialization versus OWLv2 initialization (ablation) | HALC · MSCOCO CHAIR | CHAIR_S: 23.4 -> 22.0 | Helped |
| Set exponential expanding ratio to 0.6 versus 1.0 (ablation) | HALC · MSCOCO CHAIR | CHAIR_S: 26.0 -> 22.0 | Helped |
| Set exponential expanding ratio to 0.4 instead of 0.6 (ablation) | HALC · MSCOCO CHAIR | CHAIR_S: CHAIR_S 22.0 -> 18.0 but BLEU 16.40 -> 16.10 and OPOPE 88.20 -> 87.33 | Mixed |
| Increase beam size from 2 to 8 (with same HALC settings) (ablation) | HALC · MSCOCO CHAIR | CHAIR_S: 22.0 -> 33.3 | Hurt |
| Use beam size 2 instead of 1 (ablation) | HALC · MSCOCO CHAIR | CHAIR_S: 36.0 -> 22.0 | Helped |
| Rank beams with BLIP visual matching score instead of random selection (ablation) | HALC · MSCOCO CHAIR | CHAIR_S: 26.6 -> 22.0 | Helped |
Surprising findings
- Woodpecker achieves the highest BLEU, but the authors attribute this to its use of ChatGPT, which they note makes the comparison unfair.
Limitations
- Decoding is roughly 2.4x slower than greedy without parallelization, and the analysis is theoretical rather than measured.
- The pilot study that motivates optimal visual contexts uses a small set of 110 manually constructed questions, and the oracle search succeeds for only 54% of questions.
- The paper concedes that the theoretical bound depends on assumed robustness and that the detector initialization is imperfect.
Experimental setup
Datasets: MSCOCO MME LLaVA-Bench
Benchmarks: CHAIR POPE OPOPE MME LLaVA-Bench BLEU
Metrics: CHAIR_S CHAIR_I BLEU accuracy precision F-beta MME score
Baselines: Greedy Beam Search DoLa OPERA VCD Woodpecker LURE
Builds on: contrastive decoding DoLa Grounding DINO BLIP POPE
Closest papers by meaning
- MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation (ICLR 2025 · similarity 0.91)
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.91)
- Mitigating Object Hallucination in Large Vision-Language Models via False Discovery Controlled Visual Data Splitting (NeurIPS 2026 · similarity 0.91)
- SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.91)
- Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.91)
- Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.