Research map / Vision-language and multimodal / Human and embodied modeling
Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies a few attention heads (localization heads) in frozen LVLMs whose text-to-image attention maps localize referred objects, and uses them to produce training-free REC boxes and RES masks.
Problem
Visual grounding with LVLMs usually requires fine-tuning and extra components (box or [SEG] decoders); it is unclear whether frozen LVLMs already localize objects via attention.
Approach
Select a handful of attention heads using attention-sum and spatial-entropy criteria plus a selection-frequency statistic, then aggregate their last-text-token-to-image attention maps into a pseudo-mask, bounding box, or SAM prompt, with no training.
Key results
- Training-free LLaVA-1.5-13B with 3 localization heads reaches 87.2% val REC Acc@0.5 on RefCOCO, comparable to fine-tuned Ferret-13B (89.5) and Shikra-13B (87.8).
- On RES, LLaVA-1.5-13B (ours) gets 76.1 RefCOCO val cIoU, outperforming the training-free baselines (best prior 35.2) and approaching LISA-7B (74.1).
- Using k=3 localization heads was best on average across ten LVLMs in the RES ablation (average 67.1), with k=5 degrading performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the top 3 localization heads for training-free grounding | LLaVA-1.5-13B · RefCOCO val (REC) | Acc@0.5: 92.8 -> 87.2 | Hurt |
| Use the top 3 localization heads for training-free grounding | LLaVA-1.5-13B · RefCOCO val (RES) | cIoU: 83.6 -> 76.1 | Hurt |
| Use localization heads instead of fine-tuned LISA | LLaVA-1.5-13B · ReasonSeg val overall | cIoU: 60.3 -> 60.5 | No clear effect |
| Use localization heads instead of fine-tuned LISA | LLaVA-1.5-13B · ReasonSeg test overall | cIoU: 50.8 -> 49.9 | Hurt |
| Increase localization heads from 1 to 3 (ablation) | LLaVA-1.5-13B · RefCOCO val (RES) | cIoU: 71.7 -> 76.1 | Helped |
| Increase localization heads from 3 to 4 (ablation) | LLaVA-1.5-13B · RefCOCO val (RES) | cIoU: 76.1 -> 76.0 | Hurt |
| Increase localization heads from 3 to 5 (ablation) | LLaVA-1.5-13B · RefCOCO val (RES) | cIoU: 76.1 -> 65.7 | Hurt |
| Use attention sum alone instead of both selection criteria (ablation) | LLaVA-1.5-13B localization-head selection · RefCOCO val (REC) | Acc@0.5: 87.2 -> 67.4 | Hurt |
| Use attention sum alone instead of both selection criteria (ablation) | LLaVA-1.5-13B localization-head selection · RefCOCO val (RES) | cIoU: 76.1 -> 63.8 | Hurt |
| Use spatial entropy alone instead of both selection criteria (ablation) | LLaVA-1.5-13B localization-head selection · RefCOCO val (REC) | Acc@0.5: 87.2 -> 31.3 | Hurt |
| Use spatial entropy alone instead of both selection criteria (ablation) | LLaVA-1.5-13B localization-head selection · RefCOCO val (RES) | cIoU: 76.1 -> 25.7 | Hurt |
| Select top-k heads per sample rather than using fixed heads (ablation) | LLaVA-1.5-13B localization-head selection · RefCOCO val (REC) | Acc@0.5: 87.2 -> 23.9 | Hurt |
Surprising findings
- Only three attention heads out of thousands suffice for competitive grounding performance.
- Greedy per-sample head selection performs much worse than the fixed frequency-based selection.
Limitations
- Performance on RES and REC still trails the strongest fine-tuned specialist models such as UNINEXT and PSALM.
- Localization heads may fail to distinguish similar objects, as in the failure case where attention covers multiple bananas.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg ReasonSeg
Benchmarks: RefCOCO RefCOCO+ RefCOCOg ReasonSeg
Metrics: Acc@0.5 cIoU IoU
Baselines: MDETR SeqTR G-DINO ONE-PEACE UNINEXT Shikra Ferret CogVLM ReCLIP GroundVLP LISA GSVA
Builds on: SAM LISA Gaussian smoothing
Closest papers by meaning
- GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.93)
- Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.92)
- Mechanisms of Object Localization in Vision-Language Models (CVPR 2026 · similarity 0.92)
- Look Around Before Locating: Considering Content and Structure Information for Visual Grounding (AAAI 2025 · similarity 0.91)
- Referencing Where to Focus: Improving Visual Grounding with Referential Query (NeurIPS 2024 · similarity 0.91)
- Improved Visual Grounding through Self-Consistent Explanations (CVPR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.