Research map / Architectures and efficiency / Attention and transformers
When Attention Sink Emerges in Language Models: An Empirical View
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically studies when attention sink emerges during LM pre-training, finding it depends on training steps, data, loss/weight decay, and softmax normalization, and disappears with sigmoid attention without normalization up to 1B parameters.
Problem
Attention sink, where autoregressive LMs place large attention on the first token regardless of semantics, is widely exploited but poorly understood; it is unclear when and why it emerges during pre-training.
Approach
Shows attention sink acts as key biases storing excess attention scores, emerges during pre-training after effective optimization, and is (at least partly) caused by softmax's sum-to-one dependence among attention scores; removing that dependence (sigmoid attention without normalization) prevents emergence.
Key results
- Attention sink appears in small LMs (e.g., Pythia-14M) and with random token inputs; with repeated tokens sink vanishes for Mistral-7B, LLaMA2-7B and LLaMA3-8B (Sink_1 = 0.00%).
- Replacing softmax with sigmoid attention without normalization removes attention sink at 1B parameters (Sink_1 drops from 45.11% to 2.46%) with validation loss 3.10 vs 3.07 for softmax.
- Weight decay of 0.1 gives 18.18% sink versus 41.08% at 0.5, with valid loss rising at larger decay; with 50M-token training sink disappears.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on random-token sequences instead of natural language | GPT2-XL · Random input | Sink₁ᵋ (%): 77.00 -> 70.29 | Hurt |
| use repeated-token sequences instead of natural language | Mistral-7B, LLaMA2-7B Base, and LLaMA3-8B Base · Repeated-token input | Sink₁ᵋ (%): 97.49 -> 0.00; 92.47 -> 0.00; 99.02 -> 0.00 | Hurt |
| instruction-tune the base model | Mistral-7B · Base vs Chat | Sink₁ᵋ (%): 97.49 -> 88.34 | No clear effect |
| replace Rotary with another positional embedding (ablation) | 60M LLaMA model · Pile-CC validation | Sink₁ᵋ (%): Rotary 18.18; NoPE 20.35; Absolute PE 32.73; Learnable PE 33.13; Relative PE 35.53; ALiBi 20.78 | No clear effect |
| replace pre-norm with post-norm (ablation) | 60M LLaMA model · Pile-CC validation | Sink₁ᵋ (%): 13.54% | No clear effect |
| add key biases and set value biases to zero (ablation) | 60M LLaMA model · Pile-CC validation | Sink₁ᵋ (%): 18.18 -> 0.04 | Hurt |
| replace normalized softmax with unnormalized sigmoid attention (ablation) | 60M LLaMA model · Pile-CC validation | Sink₁ᵋ (%): 18.18 -> 0.44$^*$ | Hurt |
| normalize sigmoid attention scores (ablation) | 60M LLaMA model · Pile-CC validation | Sink₁ᵋ (%): 0.44$^*$ -> 30.24 | Helped |
| replace softmax with sigmoid attention without normalization (ablation) | 1B LM · 1B model | Sink₁ᵋ (%): 45.11% -> 2.46% | Hurt |
| increase weight decay from 0.1 to 0.5 (ablation) | 60M LLaMA model · Pile-CC validation | Sink₁ᵋ (%): 18.18 -> 41.08 | Helped |
| increase weight decay further to 2.0 (ablation) | 60M LLaMA model · Pile-CC validation | valid loss: 3.73 -> 4.23 | Hurt |
| use a smaller training-data amount (ablation) | 60M LLaMA model · Pile-CC pre-training | Sink₁ᵋ | Hurt |
Surprising findings
- Attention sink persists in LMs fed random token sequences and in models with learnable sink tokens removed, shifting to other fixed positions when the data is modified.
- Re-sampling the first token randomly during pre-training increases sink (27.03%) rather than reducing it.
Limitations
- Pre-training experiments use small ~60M-parameter LLaMA-style models; large-scale claims rely on scaling to 1B only.
- Sink metric uses an empirical threshold (epsilon=0.3) without principled optimum.
- Does not establish whether attention sink benefits downstream performance.
Experimental setup
Datasets: The Pile Pile-CC validation
Benchmarks: Pile-CC validation loss HellaSwag
Metrics: Sink_k^epsilon (threshold-based attention sink metric) validation loss l2-norm of hidden states/keys/values
Baselines: softmax attention learnable sink token KV biases ReLU attention
Builds on: attention sink (StreamingLLM) massive activations linear attention softmax-off-by-one
Closest papers by meaning
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.94)
- Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings (NeurIPS 2025 · similarity 0.91)
- On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs (NeurIPS 2026 · similarity 0.90)
- Efficient Streaming Language Models with Attention Sinks (ICLR 2024 · similarity 0.89)
- From Attention to Activation: Unraveling the Enigmas of Large Language Models (ICLR 2025 · similarity 0.89)
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.