Research map / Vision-language and multimodal / Multimodal LLMs

To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that high-norm ViT attention sinks propagate into LVLM LLMs and carry coarse global semantics, then proposes a training-free sink-to-the-front reordering and a trained DIYSink (dual-MLP projector plus CoT or reweighting token selection) that improves visual reasoning benchmarks.

Problem

Existing work treats visual attention sinks as harmful and mostly within the LLM; it is unclear whether high-norm ViT sink tokens propagate into the LLM, what information they carry, and how to exploit them for LVLM visual reasoning.

Approach

Identify ViT sinks by feature norm, show they propagate to the LLM and encode global semantics useful for global tasks but harmful for local ones; reorder sinks to the front of the visual sequence (training-free), and use separate MLP projectors for sink and non-sink tokens with a learned or CoT-based selection between them.

Key results

• Sink-to-the-front improves MathVista by 1.8 points on Molmo-7B-D (training-free) and MME by 19.17 on InternVL2.5-4B.
• DIYSink (ReW) on TinyLLaVA-3B (SigLIP+Qwen2.5-3B) raises LLaVA-eval average from 70.22 to 71.65 and MathVista from 30.40 to 33.10.
• DIYSink (ReW) on TinyLLaVA-3B (SigLIP+Phi-2) improves LLaVA-eval average by 5.79 points (48.55 to 54.34).

Surprising findings

• Sink tokens propagated from ViT receive attention comparable to or higher than LLM-emerged sinks in LLaVA-7B (1.13% vs 1.27%, versus 0.15% for non-sinks).
• Sink-only inputs outperform non-sink-only on global tasks while removing sinks helps local tasks, so the two token types have opposite task preferences.

Limitations

• Gains in training-based setting are on small TinyLLaVA models and LLaVA-7B; improvements on some benchmarks are small or negative.
• Sink thresholds are set per backbone by hand and the DeepSeek-VL channel-concatenated features complicate the sink behavior, giving smaller gains.

Experimental setup

Datasets: LLaVA training sets PixMo GeoQA COCO (linear probing)

Benchmarks: LLaVA eval GQA ScienceQA TextVQA MMMU MME MathVista

Metrics: accuracy MME score

Baselines: LLaVA-7B TinyLLaVA InternVL2.5-4B Phi 3.5 V DeepSeek-VL-7B Molmo-7B-D

Builds on: attention sinks register tokens LLaVA

Closest papers by meaning

• See What You Are Told: Visual Attention Sink in Large Multimodal Models (ICLR 2025 · similarity 0.92)
• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.91)
• When to Think and When to Look: Uncertainty-Guided Lookback (CVPR 2026 · similarity 0.91)
• Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models (ICLR 2026 · similarity 0.90)
• Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas (ICML 2025 · similarity 0.90)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.90)