Research map / Large language models / Language, safety and interpretability
Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Investigates attention sinks in LLMs, proposing a "catch, tag, release" mechanism where sinks catch tokens, tag them via value vectors, and release them for later retrieval, with probing and a minimal sequence-averaging theory.
Problem
Attention sinks beyond the first token (prompt-dependent sinks on punctuation) are poorly understood: their function, semantic role, and origin remain unclear, including whether they differ in reasoning models and under query-key normalization.
Approach
Attention sinks catch subsequent tokens, tag them with a shared direction in value space, and release them into the residual stream where deeper layers retrieve tokens by their tags; tags carry semantic information (e.g., truth of a statement) and a minimal averaging task shows this mechanism emerges via training.
Key results
- Tag-component mass-mean probes reach 98% (Qwen 2.5 3B), 94.5% (Qwen 2.5 7B), 99.5% (Qwen 2.5 14B), and 99.0% (LLaMA-3 8B) accuracy on cities true/false, while non-tag probes stay near 50%.
- Tags typically explain 20%-40% of attention head output variance, and up to 70% in many cases.
- Reasoning-distilled DeepSeek models show more sinks and more heads with high variance explained by tags than pretrained counterparts, and QK-normalized models show similar sink counts to those without.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the tag component instead of the non-tag component | Mass-mean probes · Cities validation across Qwen2.5 and LLaMA models | classification accuracy: 50.0% -> 98%; 50.0% -> 94.5%; 50.5% -> 99.5%; 56.5% -> 99.0%; 50.0% -> 92.5% | Helped |
| Use the tag component instead of the full activation | Mass-mean probes · Cities validation across Qwen2.5 and LLaMA models | classification accuracy: 50.0% -> 98%; 83.0% -> 94.5%; 60% -> 99.5%; 97.0% -> 99.0%; 86.0% -> 92.5% | Mixed |
| Fine-tune for reasoning with DeepSeek-R1 distillation | Pretrained LLaMA 3.1 8B and Qwen2.5 14B · Attention sinks and tag-explained variance across heads and layers | attention sink count and variance explained | Helped |
| Apply QK normalization | Models with and without query-key normalization · Attention sink formation | average number of sinks | No clear effect |
| Lower the sink-identification threshold (ablation) | Sink and tag measurement across tested models · Threshold sensitivity analysis | average variance explained by tags | Mixed |
Surprising findings
- Tag-component probes can outperform full-activation probes (e.g., 98% vs 50% for Qwen 2.5 3B), suggesting tags give a denoised truth direction.
- Query-key normalization does not eliminate attention sink formation, with sink counts remaining similar to models without it.
Limitations
- Mechanism is demonstrated via a single representative example and aggregated statistics on few prompts; generality beyond tested models is not established.
- Theory covers a minimal sequence-averaging toy task rather than natural language behavior.
- Threshold epsilon=0.2 for sink identification is a chosen hyperparameter whose sensitivity is only analyzed in the appendix.
Experimental setup
Datasets: cities dataset QuALITY 170 prompts (Ben-style prompt collection) sequence averaging synthetic task
Benchmarks: cities dataset
Metrics: probe classification accuracy variance explained by tags attention sink count
Baselines: full-activation probe non-tag probe
Builds on: attention sinks (StreamingLLM) mass-mean probing DeepSeek-R1 distillation QK normalization
Closest papers by meaning
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.91)
- When Attention Sink Emerges in Language Models: An Empirical View (ICLR 2025 · similarity 0.91)
- On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs (NeurIPS 2026 · similarity 0.90)
- Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs (AAAI 2025 · similarity 0.88)
- OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference (ICML 2025 · similarity 0.88)
- Efficient Streaming Language Models with Attention Sinks (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.