Research map / Vision-language and multimodal / Multimodal LLMs
Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dropout Decoding, an inference-time method that projects visual tokens into text space, measures their epistemic uncertainty, and masks high-uncertainty visual tokens to ensemble majority-voted decodes that reduce object hallucination in LVLMs.
Problem
LVLMs misinterpret visual inputs, causing object hallucinations and unreliable outputs; existing fixes need training or use heuristic inference-time methods with extra cost.
Approach
Quantify each visual token's epistemic uncertainty as the KL divergence between its logit-lens text-space projection and the average projection over all visual tokens, then randomly drop visual tokens with probability tied to that uncertainty and majority-vote over K masked decoding contexts.
Key results
- On InstructBLIP, Dropout Decoding reduces CHAIR_I to 6.63 from 7.90 (greedy) and CHAIR_S to 24.53 from 27.87.
- On MMBench, Dropout Decoding reaches 74.01 on LLaVA-1.5 vs 71.86 for original decoding, the best among the compared methods on that model.
- Masking high-confidence tokens instead of high-uncertainty ones yields results close to greedy decoding, supporting the uncertainty-guided choice.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Dropout Decoding rather than the strongest listed baseline | LLaVA-1.5 · MSCOCO CHAIR | CHAIR_S: 41.47$_{\pm 0.92}$ -> 39.80$_{\pm 2.3}$ | Helped |
| Apply Dropout Decoding rather than the strongest listed baseline | InstructBLIP · MSCOCO CHAIR | CHAIR_S: 25.87$_{\pm 2.77}$ -> 24.53$_{\pm 1.26}$ | Helped |
| Apply Dropout Decoding rather than the strongest listed baseline | LLaVA-NEXT · MSCOCO CHAIR | CHAIR_I: 7.10$_{\pm 0.20}$ -> 7.39$_{\pm 0.69}$ | Hurt |
| Apply Dropout Decoding rather than the strongest listed baseline | LLaVA-1.5 · MSCOCO THRONE | F1_all: 0.802$_{\pm 0.003}$ -> 0.804$_{\pm 0.002}$ | Helped |
| Apply Dropout Decoding rather than the strongest listed baseline | LLaVA-1.5 · MMBench | score: 73.86 -> 74.01 | Helped |
| Apply Dropout Decoding rather than the strongest listed baseline | LLaVA-NEXT · MMBench | score: 74.57 -> 74.31 | Hurt |
| Omit the preliminary forward pass (ablation) | Dropout Decoding on LLaVA-1.5 · MSCOCO CHAIR | CHAIR_S: 39.80$_{\pm 2.3}$ -> 39.73$_{\pm 2.15}$ | No clear effect |
| Replace uncertainty-guided masking with random masking (ablation) | LLaVA-1.5 · MSCOCO CHAIR | CHAIR_S: 39.80$_{\pm 2.3}$ -> 35.93$_{\pm 0.90}$ | Helped |
| Replace uncertainty-guided masking with random masking (ablation) | LLaVA-1.5 · MSCOCO CHAIR | BLEU: 11.64$_{\pm 0.12}$ -> 11.51$_{\pm 0.12}$ | Hurt |
| Mask high-confidence rather than low-confidence visual tokens (ablation) | Dropout Decoding · MSCOCO CHAIR and THRONE | CHAIR_S | Mixed |
Surprising findings
- Random masking reduces CHAIR scores but produces repetitive outputs (e.g., 'apple apple apple'), artificially inflating CHAIR and failing to compute THRONE.
- Omitting the preliminary forward pass works comparably on LLaVA models, suggesting models with many visual tokens may not need it.
Limitations
- Extra forward passes add decoding overhead, especially with the preliminary pass.
- Epistemic uncertainty estimates depend on the quality of the text-space projections, which vary with the projector.
- Evaluated only on object hallucination benchmarks and MMBench, with a single prompt type for captioning.
Experimental setup
Datasets: MSCOCO 2014
Benchmarks: CHAIR THRONE MMBench
Metrics: CHAIR_S CHAIR_I F1_all F0.5_all P_all R_all BLEU accuracy
Baselines: Greedy Beam Search OPERA VCD
Builds on: VCD OPERA logit lens MC dropout
Closest papers by meaning
- On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models (NeurIPS 2025 · similarity 0.92)
- When to Think and When to Look: Uncertainty-Guided Lookback (CVPR 2026 · similarity 0.90)
- VES-RFT: Rewarding Visual Evidence Sensitivity to Mitigate Hallucinations in Large Vision-Language Models (CVPR 2026 · similarity 0.90)
- Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.90)
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (CVPR 2024 · similarity 0.90)
- Mitigating Object Hallucination in Large Vision-Language Models via False Discovery Controlled Visual Data Splitting (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.