Research map / Vision-language and multimodal / Multimodal LLMs

AFTER: Mitigating the Object Hallucination of LVLM via Adaptive Factual-Guided Activation Editing

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AFTER, an inference-time activation editing method that steers LVLM attention-head activations toward factual textual descriptions built from COCO ground-truth annotations, with a lightweight query-aware MLP estimating per-query offsets, reducing object hallucination on POPE, MME and AMBER.

Problem

LVLMs suffer object hallucination (category, attribute, relation) caused by language bias; existing activation editing methods use visually degraded images as negative guidance and apply one identical steering vector to all queries.

Approach

Build trusted activations from factual text (ground-truth annotations converted into category, attribute, and relation facts and textualized by an LVLM/LLM) rather than degraded images, derive a general steering vector from them, and train a single-layer MLP offset estimator to produce query-specific edits added to that vector.

Key results

• On AMBER, AFTER reduces hallucination (Hal) by up to 16.3% over baseline on Shikra (49.5 to 33.2 is reported as the headline reduction in the paper's text).
• On POPE, average accuracy improves 4.1% and F1 2.6% over baselines across three LVLMs, surpassing ICT by 1.3% and 0.9%.
• Generalization: on COCO->GQA POPE, LLaVA-v1.5 accuracy rises from 76.9 to 84.6.

Surprising findings

• Simple COCO captions as direct input perform 6.7% worse than the raw image on POPE.
• Using factual descriptions from a weaker LVLM (LLaVA-v1.5 7B) gives nearly the same steering performance as GPT-4o, suggesting the textualizer is not a knowledge source.

Limitations

• Depends on access to LLM/LVLM activations of open-source models, so it cannot apply to closed-source models.
• Needs domain-specific data for specialized domains such as medical report analysis.

Experimental setup

Datasets: COCO train COCO A-OKVQA GQA

Benchmarks: POPE MME AMBER

Metrics: accuracy F1 MME score CHAIR Hal Cover

Baselines: Baseline HACL VCD OPERA VTI ICT

Builds on: VTI ICT inference-time activation steering

Closest papers by meaning

• AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM (CVPR 2026 · similarity 0.92)
• HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Models (CVPR 2026 · similarity 0.92)
• Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models (ICLR 2026 · similarity 0.92)
• Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance (ICML 2025 · similarity 0.92)
• Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning (NeurIPS 2025 · similarity 0.92)
• Hallucination-aware Intermediate Representation Editing in Large Vision-Lanugage Models (ICLR 2026 · similarity 0.92)