Research map / Vision-language and multimodal / Multimodal LLMs

Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GACD, a training-free inference-time decoding method that estimates per-token visual/text influence via gradient norms and uses influence-constrained contrastive decoding plus early stopping to reduce MLLM hallucinations.

Problem

MLLMs hallucinate objects not grounded in the image, driven by text-visual bias (over-reliance on prompts and prior outputs) and co-occurrence bias (spurious object correlations), and existing mitigation methods rely on auxiliary models, heuristic priors, or training.

Approach

Estimate token influence via first-order Taylor gradient L1 norms, group visual tokens into object-related and unrelated sets using the top-influence visual token of each previously emitted noun, then contrastively decode with a closed-form alpha that rebalances visual influence to match the dominant text influence.

Key results

• On AMBER with LLaVA-v1.5, GACD raises score from 83.5 to 90.2 and discriminative accuracy from 72.0 to 80.3 while lowering hallucination rate (hal) from 36.4 to 24.3.
• On POPE MSCOCO adversarial with mPLUG-Owl2, accuracy/F1 rise from 72.5/77.5 to 84.2/83.7.
• On the MSCOCO CHAIR subset with LLaVA-v1.5, C_S drops from 48.8 to 41.0 and C_I from 13.4 to 10.9, but recall also falls slightly and length drops from 99.8 to 85.0.

Surprising findings

• Co-occurrence hallucinated objects often share the same most influential visual token as the true co-occurring object (31.9% of cases in single-object images).
• Early stopping and CHAIR gains can come at the cost of recall; the authors note that a 12% ES threshold yields recall of 55.0 versus 78.6 baseline on LLaVA-v1.5.

Limitations

• Requires white-box access to gradients, so it cannot be applied to closed API models.
• Effectiveness depends on the baseline model's original visual influence ratio and on whether the question relies on visual information.
• Early stopping trades recall for lower hallucination; performance on MM-Vet generalization and reasoning categories is mixed.

Experimental setup

Datasets: AMBER MSCOCO LLaVA-QA90 POPE MMBench MM-Vet ScienceQA VizWiz MME

Benchmarks: AMBER MSCOCO CHAIR subset LLaVA-QA90 POPE MMBench MM-Vet ScienceQA VizWiz MME

Metrics: CHAIR (C_S, C_I) recall coverage hallucination rate co-occurrence hallucination accuracy F1 detailedness score

Baselines: RLAIF-V VCD M3ID AVISC Woodpecker PAI Middle

Builds on: visual contrastive decoding M3ID AVISC Integrated Gradients

Closest papers by meaning

• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
• Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models (ICLR 2025 · similarity 0.93)
• ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models (AAAI 2025 · similarity 0.92)
• Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating (ICML 2026 · similarity 0.92)
• Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.92)
• Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding (CVPR 2025 · similarity 0.92)