Research map / Vision-language and multimodal / Multimodal LLMs

Hallucination-aware Intermediate Representation Editing in Large Vision-Lanugage Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HIRE, a training-light feature-editing framework that detects hallucination-prone attention representations in LVLMs and edits them via a learned Editor (dual-encoder autoencoder) and a Router, with a signed edit strength to both suppress and amplify hallucinations.

Problem

LVLMs produce object hallucinations that contradict visual content; existing fixes either require costly retraining or contrastive decoding that doubles inference cost and applies uniform adjustments to all tokens, and offer little controllable hallucination level.

Approach

Learn an Editor that separates semantic and hallucination subspaces of attention-layer representations (contrastive + reconstruction losses) and produces a token-specific editing direction, plus a lightweight Router (trained with a DPO-style objective) that decides from the first layer whether to edit, with a signed strength alpha for controllable hallucination.

Key results

• On CHAIR with LLaVA-1.5 (512 tokens), HIRE reduces CHAIR_S from 51.3 to 30.2 and CHAIR_I from 16.8 to 9.7 at 11.81 TFLOPs vs 20.46-21.39 for CD baselines.
• On POPE ALL, HIRE with LLaVA-1.5-7B reaches 87.27 accuracy / 87.23 F1 vs 82.04 / 80.42 baseline.
• On AMBER, HIRE raises the AMBER score of LLaVA-1.5 from 81.58 to 89.12, the best among compared methods (Octopus 88.95).

Surprising findings

• Editing only the first-layer-based router decision with a single shared decision across layers matches per-layer routers in hallucination reduction while cutting training time.
• Editing deep layers (20-29) contributes little to hallucination reduction, while middle layers matter most.

Limitations

• Trains on all tokens and layers, which can be noisy; selective training on critical hidden states is left to future work.
• Evaluated mainly on object hallucination benchmarks (CHAIR, POPE, AMBER) with limited model variety.
• Hallucination control is shown only via a scalar edit strength; generalization of controllability is shown mostly on CHAIR.

Experimental setup

Datasets: MSCOCO Visual Genome

Benchmarks: CHAIR POPE AMBER MME SEED-Bench GQA A-OKVQA

Metrics: CHAIR_S CHAIR_I Accuracy F1 AMBER score TFLOPs BLEU R^2

Baselines: VCD ICD M3ID AVISC OPERA SID Octopus VTI Projectaway Nullu HA-DPO HACL

Builds on: Direct Preference Optimization Contrastive Decoding (VCD) InfoNCE CHAIR

Closest papers by meaning

• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.94)
• HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Models (CVPR 2026 · similarity 0.93)
• Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations (ICLR 2025 · similarity 0.93)
• SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding (ICML 2025 · similarity 0.93)
• Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling (NeurIPS 2025 · similarity 0.93)
• Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models (ICLR 2026 · similarity 0.93)