Research map / Vision-language and multimodal / Multimodal LLMs
Heads That Write, Not Just Point: Image Retrieval Heads in Vision-Language Models
Junsung Park, Uiwon Hwang, Donghun Kang, Yeongtak Oh, Jaewon Jeong, Gyeongtae Yoo, Han Cheol Moon, Jungbeom Lee, Sungroh Yoon
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how large vision-language models identify the relevant image in multi-image contexts by analyzing per-head attention over image-token spans, using a controlled probe called ICIR-MCQ.
Problem
How LVLMs internally identify the image relevant to a question when multiple interleaved images are in context remains poorly understood.
Approach
Frames the grounding step as in-context image retrieval and analyzes image retrieval at attention-head granularity, using each head's attention mass over image-token spans, via a controlled multiple-choice probe (ICIR-MCQ).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use pointer-writer overlap heads for ablation vs single-criterion heads (ablation) | LVLM head ablation (top-K zero-out) · Visual Haystacks N=2-20 | accuracy drop | Improved |
| ablate writer-only heads (ablation) | LVLM head ablation · Visual Haystacks | accuracy drop | No clear effect |
| ablate pointer-only heads (ablation) | LVLM head ablation · Visual Haystacks | accuracy drop | Mixed |
| use single pointer head instead of external retrievers | L21H10 pointer head · Visual Haystacks N=10 | R@1: .649 (SigLIP) -> .832 (Qwen3-VL-8B head) | Improved |
| rank heads by output-side writer criterion instead of attention pointer criterion | head discovery · ICIR-MCQ | top-100 head overlap: 35-47 heads overlap | Mixed |
| increase ablation budget K and haystack size N (ablation) | pointer-writer ablation · Visual Haystacks | accuracy drop | Mixed |
| compare 4B and 8B models for pointer retrieval | pointer head · Visual Haystacks N=10 | R@1: .794/.836 (4B) vs .787/.832 (8B) | No clear effect |
Experimental setup
Benchmarks: ICIR-MCQ
Metrics: attention mass over image-token spans
Closest papers by meaning
- What's in the Image? A Deep-Dive into the Vision of Vision Language Models (CVPR 2025 · similarity 0.87)
- Leveraging Large Vision-Language Model as User Intent-Aware Encoder for Composed Image Retrieval (AAAI 2025 · similarity 0.87)
- Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration (ICML 2026 · similarity 0.87)
- GenIR: Generative Visual Feedback for Mental Image Retrieval (NeurIPS 2025 · similarity 0.87)
- Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads (NeurIPS 2026 · similarity 0.87)
- MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
