Research map / Vision-language and multimodal / Multimodal LLMs

Look Closer! An Adversarial Parametric Editing Framework for Hallucination Mitigation in VLMs

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ALEAHallu, an activate-locate-edit adversarial parametric editing method that localizes the second MLP projection in the most divergent layer of a VLM and edits it with an adversarially prefixed prompt to reduce object hallucination.

Problem

Object hallucination in vision-language models, where generated text conflicts with the image, driven by over-reliance on language priors and weak visual grounding; existing decoding fixes are non-trainable and costly.

Approach

Build positive (visually grounded) and negative (prior-driven) description pairs, locate the layer with largest hidden-state distance, and edit that layer's MLP down-projection so the model answers correctly under a learnable adversarial prefix that pushes toward visual neglect, with a KL constraint preserving benign behavior; no extra inference cost.

Key results

• On LLaVA-1.5 image captioning, ALEAHallu reduces CHAIR_s from 56.4 (regular) to 39.3 and CHAIR_i from 17.2 to 12.1, with recall 74.2.
• On POPE MSCOCO random/popular/adversarial accuracy, ALEAHallu reaches 0.9072/0.8952/0.8016 versus 0.8452/0.8288/0.8076 for regular decoding.
• Average attention to image tokens rises from 29.71% to 33.72% on 500 test samples.

Surprising findings

• Beam search and OPERA achieve higher recall than ALEAHallu on LLaVA-1.5 (78.8 vs 74.2), and Nullu beats ALEAHallu on some POPE adversarial and A-OKVQA random settings.
• Random layer editing in the ablation performs much worse than the proposed localization, suggesting that layer choice matters.

Limitations

• Recall lower than beam search on LLaVA-1.5 captioning
• Evaluation relies on GPT-4o for labeling and small training sets (~2k pairs)
• Mixed gains: ALEAHallu does not win every POPE setting

Experimental setup

Datasets: MSCOCO A-OKVQA GQA

Benchmarks: CHAIR POPE MME

Metrics: CHAIR_s CHAIR_i Recall caption length Accuracy Precision F1 score

Baselines: Regular decoding Beam Search OPERA VCD Nullu

Builds on: Nullu VCD OPERA locate-and-edit (ROME/MEMIT-style)

Closest papers by meaning

• Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs (NeurIPS 2025 · similarity 0.94)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
• One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination (CVPR 2026 · similarity 0.92)
• Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback (AAAI 2025 · similarity 0.92)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.92)
• Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations (ICLR 2025 · similarity 0.92)