Research map / Vision-language and multimodal / Multimodal LLMs

Envision, Attend, Then Respond: Counterfactual Hallucination Mitigation in Large Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EnAR, a training-free Envision-Attend-Respond framework that builds a diffusion-based visual impression, localizes counterfactual tokens via vision-encoder attention and uncertainty, pads them, and contrastively decodes to reduce counterfactual and general hallucinations in LVLMs.

Problem

LVLMs hallucinate when visual evidence conflicts with world knowledge (counterfactual scenarios, e.g., a five-legged llama answered as four legs), and existing decoding-level mitigation methods fail to localize counterfactual elements.

Approach

Use a diffusion model's latent perturbation (Langevin dynamics in DDIM latent space) to produce a prior-consistent visual impression plus a pixel-wise uncertainty map; compare vision-encoder attention between original and impression to localize counterfactual tokens, replace them with pad tokens, and apply VCD-style contrastive decoding between original and padded inputs.

Key results

• On VLMBias, EnAR raises InternVL3.5-8B overall accuracy from 19.83% to 31.36% (+11.53 points, reported as 10.82% gain in abstract).
• On POPE, EnAR improves LLaVA-v1.5-7B F1 by 8.1%, 8.0%, and 6.9% across Random, Popular, and Adversarial settings.
• On WHOOPS, EnAR raises InternVL3.5-8B average accuracy from 62.45% to 74.15%.

Surprising findings

• All decoding baselines, and EnAR, score 0% on the Chess Pieces category of VLMBias across backbones.

Limitations

• Requires running a diffusion model and multiple perturbation samples per image, adding inference cost.
• Chess Pieces remains at 0% accuracy for all methods, indicating limits of the approach on some counterfactual categories.
• Relies on hyperparameters (layer 6, 10% padding ratio) selected empirically.

Experimental setup

Datasets: VLMBias PhD WHOOPS HallusionBench POPE HKU-IS

Benchmarks: VLMBias PhD WHOOPS HallusionBench POPE

Metrics: accuracy F1 precision recall aAcc fAcc qAcc

Baselines: VCD M3ID RITUAL DeGF AGLA Regular decoding

Builds on: VCD DDIM Langevin dynamics Tweedie estimator

Closest papers by meaning

• Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception (CVPR 2025 · similarity 0.93)
• Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models (ICLR 2026 · similarity 0.93)
• Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating (ICML 2026 · similarity 0.93)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.93)
• Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy (ICML 2026 · similarity 0.92)
• Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation (ICLR 2026 · similarity 0.92)