Research map / Vision-language and multimodal / Multimodal LLMs

Fighting Hallucinations with Counterfactuals: Diffusion-Guided Perturbations for LVLM Hallucination Suppression

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CIPHER, a training-free test-time method that builds a low-rank hallucination subspace from diffusion-edited counterfactual images (OHC-25K) and projects LVLM hidden states away from it to reduce vision-induced hallucinations.

Problem

LVLMs frequently generate object hallucinations unfaithful to the visual input; existing training-free methods mostly target text-induced hallucinations, leaving vision-induced hallucinations underexplored.

Approach

Generate counterfactual images with Stable Diffusion conditioned on GPT-perturbed captions, contrast LVLM hidden states of counterfactual vs. authentic image-caption pairs, extract the top singular vectors as a hallucination subspace, and project hidden states onto its orthogonal complement at inference.

Key results

• On LLaVA-1.5, CIPHER reduces CHAIRS from 20.40 (greedy) to 13.05 and beats the second-best Nullu (15.20).
• On MiniGPT-4, CHAIRS drops to 18.48 vs 32.40 greedy; on mPLUG-Owl2 to 13.60 vs 22.90 greedy.
• On LLaVA-Bench with GPT-4V judging, CIPHER improves LLaVA-1.5 accuracy from 6.79 to 7.08 and detailedness from 6.33 to 6.75.

Surprising findings

• Combining text and image hallucination sources degrades performance (CHAIRS 15.71) relative to image-only (13.05).
• Text-induced perturbation subspaces give only moderate, unstable linear separability across layers compared with diffusion-based visual perturbations.

Limitations

• Fixed offline projection subspace; not adapted per input.
• Evaluation on a small subset (500 images) for CHAIR and POPE; the paper reports mean and std over three seeds.
• Rank r and projection layers selected via grid search per model.

Experimental setup

Datasets: MSCOCO OHC-25K

Benchmarks: CHAIR Offline POPE MMHal-Bench LLaVA-Bench

Metrics: CHAIRS CHAIRI BLEU accuracy precision F1 GPT-4V score

Baselines: Greedy Beam Search DoLa OPERA VCD Woodpecker LURE HALC Nullu

Builds on: Nullu LURE OPERA VCD

Closest papers by meaning

• One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination (CVPR 2026 · similarity 0.92)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.91)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.91)
• Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations (CVPR 2026 · similarity 0.91)
• GHOST: Hallucination-Inducing Image Generation for Multimodal LLMs (ICLR 2026 · similarity 0.91)
• Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.90)