AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

Heads That Write, Not Just Point: Image Retrieval Heads in Vision-Language Models

Junsung Park, Uiwon Hwang, Donghun Kang, Yeongtak Oh, Jaewon Jeong, Gyeongtae Yoo, Han Cheol Moon, Jungbeom Lee, Sungroh Yoon

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies how large vision-language models identify the relevant image in multi-image contexts by analyzing per-head attention over image-token spans, using a controlled probe called ICIR-MCQ.

Problem

How LVLMs internally identify the image relevant to a question when multiple interleaved images are in context remains poorly understood.

Approach

Frames the grounding step as in-context image retrieval and analyzes image retrieval at attention-head granularity, using each head's attention mass over image-token spans, via a controlled multiple-choice probe (ICIR-MCQ).

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use pointer-writer overlap heads for ablation vs single-criterion heads (ablation)LVLM head ablation (top-K zero-out) · Visual Haystacks N=2-20accuracy dropImproved
ablate writer-only heads (ablation)LVLM head ablation · Visual Haystacksaccuracy dropNo clear effect
ablate pointer-only heads (ablation)LVLM head ablation · Visual Haystacksaccuracy dropMixed
use single pointer head instead of external retrieversL21H10 pointer head · Visual Haystacks N=10R@1: .649 (SigLIP) -> .832 (Qwen3-VL-8B head)Improved
rank heads by output-side writer criterion instead of attention pointer criterionhead discovery · ICIR-MCQtop-100 head overlap: 35-47 heads overlapMixed
increase ablation budget K and haystack size N (ablation)pointer-writer ablation · Visual Haystacksaccuracy dropMixed
compare 4B and 8B models for pointer retrievalpointer head · Visual Haystacks N=10R@1: .794/.836 (4B) vs .787/.832 (8B)No clear effect

Experimental setup

Benchmarks: ICIR-MCQ

Metrics: attention mass over image-token spans

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy