Research map / Vision-language and multimodal / Multimodal LLMs

Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a training-free, attention-guided debiasing method that corrects position bias in MLLM multi-image retrieval using attention priors and a 5-sample counterfactual calibration set, reporting large accuracy gains over PriDe and prompting baselines.

Problem

MLLMs in multi-image cross-modal retrieval show severe position bias, where predictions follow input order rather than semantic relevance; existing logit-level calibration (PriDe) assumes a static global bias.

Approach

Identifies Logit-Attention Divergence (attention peaks at the correct image while output logits are biased) and uses a conditional, GT-position-dependent bias estimate plus an attention-derived visual posterior, with layer selection, to debias logits per instance in a single forward pass.

Key results

• On Random N=4 with LLaVA-OneVision-8B, accuracy rises from 67.52% (Vanilla) and 68.00% (PriDe) to 98.66% with the proposed method.
• On Adversarial N=4 with Qwen2.5-VL-3B, accuracy goes from 47.80% (PriDe) to 65.72%.
• Calibrating on Flickr8k and testing on MS-COCO random yields 97.78% accuracy and 93.40% consistency on LLaVA-OneVision.

Surprising findings

• Raw attention readout is worse than vanilla logits in the adversarial N=4 setting (42.36% vs 49.56%), attributed to attention sinks at boundary positions.
• Prompting strategies such as instruction and CoT often fail to reduce position bias and sometimes underperform vanilla.

Limitations

• Requires white-box access to internal attention, so it cannot be applied to closed-source APIs.
• Gains are smaller on visually ambiguous tasks where attention is less reliable; accuracy is bounded by the model's intrinsic discrimination.
• Formulation is tailored to discriminative selection over discrete candidates and does not directly extend to open-ended generation.

Experimental setup

Datasets: MS-COCO Flickr8k

Benchmarks: MS-COCO multi-image retrieval (random and adversarial, 1,000 samples each) Flickr8k cross-domain MMIU six-task subset

Metrics: accuracy recall standard deviation consistency

Baselines: Vanilla Instruction prompting Chain-of-Thought PriDe Self-Consistency SoFA Permutation Averaging

Builds on: PriDe SoFA self-consistency

Closest papers by meaning

• Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding (CVPR 2026 · similarity 0.86)
• Identifying and Mitigating Position Bias of Multi-image Vision-Language Models (CVPR 2025 · similarity 0.86)
• Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping (ICLR 2026 · similarity 0.86)
• Scale-invariant attention (NeurIPS 2025 · similarity 0.86)
• WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.86)
• Q Cache: Visual Attention Is Valuable in Less than Half of Decode Layers for Multimodal Large Language Model (AAAI 2026 · similarity 0.86)