Research map / Vision-language and multimodal / Multimodal LLMs
See What You Are Told: Visual Attention Sink in Large Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies "visual attention sink" tokens in LMMs (massive activations in sink dimensions) and proposes VAR, a training-free method that redistributes attention mass from sink tokens to visual tokens in selected image-centric heads.
Problem
LMMs allocate high attention to visual tokens irrelevant to the text token, and insufficient attention to images overall, which may hurt visual grounding and hallucination.
Approach
Visual tokens with massive activations in the same sink dimensions as BOS tokens receive high attention but contribute little to outputs; their attention mass is recycled as a budget and redistributed to non-sink visual tokens, only in heads selected as image-centric via a visual non-sink ratio.
Key results
- VAR improves LLaVA-1.5-7B on VizWiz from 50.0 to 53.7 and MM-Vet from 31.1 to 33.7 without training.
- On MMHal-Bench, VAR raises LLaVA-1.5-7B score from 2.36 to 4.26 and lowers hallucination rate from 51.0 to 45.1.
- On MMVP, VAR raises LLaVA-1.5-7B from 3.33 to 9.33 and InternVL2-8B from 51.3 to 56.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply VAR attention redistribution | LLaVA-1.5-7B · GQA | score: 62.0 -> 63.5 | Helped |
| apply VAR attention redistribution | LLaVA-1.5-7B · VizWiz | score: 50.0 -> 53.7 | Helped |
| apply VAR attention redistribution | LLaVA-1.5-7B · MME | score: 1495.5 -> 1513.8 | Helped |
| apply VAR attention redistribution | LLaVA-1.5-7B · MM-Vet | score: 31.1 -> 33.7 | Helped |
| apply VAR attention redistribution | LLaVA-1.5-7B · CHAIR | C_S: 45.0 -> 43.2 | Helped |
| apply VAR attention redistribution | LLaVA-1.5-7B · POPE all | F1: 85.90 -> 86.53 | Helped |
| apply VAR attention redistribution | LLaVA-1.5-7B · MMHal-Bench | score: 2.36 -> 4.26 | Helped |
| apply VAR attention redistribution | LLaVA-1.5-7B · MMVP | score: 3.33 -> 9.33 | Helped |
| remove image-centric head selection (ablation) | VAR on LLaVA-1.5-7B · POPE | F1: 86.5 -> 0.0 | Hurt |
| remove image-centric head selection (ablation) | VAR on LLaVA-1.5-7B · MME | Score: 1513.8 -> 0.0 | Hurt |
| redistribute attention to text tokens only instead of visual tokens (ablation) | VAR on LLaVA-1.5-7B · POPE | F1: 85.9 -> 70.3 | Hurt |
| redistribute attention to both visual and text tokens rather than visual tokens only (ablation) | VAR on LLaVA-1.5-7B · MME | Score: 1513.8 -> 1503.9 | Hurt |
Surprising findings
- Redistributing attention to text tokens alone yields little gain or even decreases performance, while visual-only redistribution is best.
- Applying redistribution to all heads without selection drives performance to zero, so head selection is essential.
Limitations
- Uses raw visual attention maps without post-processing or object localization refinement.
- Only evaluated on a set of LMMs up to 13B-scale; applicability to larger models not explored.
- Hyperparameters (rho) vary per task type and require a small pseudo-validation set for tuning.
Experimental setup
Datasets: Pascal-VOC MS-COCO LLaVA-Bench (In-the-Wild)
Benchmarks: VQAv2 GQA VizWiz ScienceQA-IMG TextVQA MME MMBench-en SEED-Bench-image LLaVA-Bench (In-the-Wild) MM-Vet CHAIR POPE
Metrics: accuracy F1 CHAIR_S CHAIR_I MMHal score hallucination rate MME score GPT-4 score
Baselines: LLaVA-1.5-7B LLaVA-1.5-13B LLaVA-1.5-HD-13B VILA-13B Qwen2-VL-7B InternVL2-8B VCD
Builds on: attention sink (StreamingLLM) massive activations visual contrastive decoding
Closest papers by meaning
- To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models (ICLR 2026 · similarity 0.92)
- Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach (CVPR 2025 · similarity 0.91)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.90)
- Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs (NeurIPS 2026 · similarity 0.90)
- Blink: Dynamic Visual Token Resolution for Enhanced Multimodal Understanding (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.