Research map / Architectures and efficiency / Attention and transformers
Anatomy of Massive Activations and Attention Sinks
Shangwen Sun, Alfredo Canziani, Yann LeCun, Jiachen Zhu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add per-channel conditional gating (ablation) | Llama-style 7B · C4 sentences, 7B from scratch | sink ratio: 46.0% -> 4.5% | Improved |
| add static unconditional channel gating (ablation) | Llama-style 7B · C4 sentences, 7B from scratch | sink ratio: 46.0% -> 42.2% | No clear effect |
| add sandwich normalization (ablation) | Llama-style 7B · C4 sentences, 7B from scratch | spike magnitude: 3,818 -> 520 | Improved |
| add QK-Norm to sandwich normalization (ablation) | Llama-style 7B · C4 sentences, 7B from scratch | spike magnitude: 3,818 -> 92 | Improved |
| replace pre-norm with DynamicTanh (ablation) | Llama-style 7B · C4 sentences, 7B from scratch | sink ratio: 46.0% -> 61.0% | Improved |
| replace SwiGLU FFN with linear or attention-only blocks (ablation) | Llama-style 7B · C4 sentences, 7B from scratch | spike magnitude: 3,818 -> 688 | Improved |
| increase head dimension from 8 to 128 (ablation) | Llama-style 7B · C4 sentences, 7B from scratch | sink ratio: 4.1% -> 46.0% | Improved |
| train only on long-context positions 2048-4096 (ablation) | Llama-style 7B · C4 sentences, 7B from scratch | sink ratio: 46.0% -> 1.2% | Hurt |
| train with maximum context 256 instead of 4096 (ablation) | Llama-style 7B · C4 sentences, 7B from scratch | sink ratio: 46.0% -> 42.1% | No clear effect |
| disable weight decay (ablation) | Llama-style 7B · C4 sentences, 7B from scratch | spike magnitude: 3,818 -> 12,275 | Hurt |
| extend training from 100B to 200B tokens (ablation) | Llama-style 7B · C4 sentences, 7B from scratch | sink ratio: 46.0% -> 63.3% | Improved |
| raise AdamW beta2 from 0.95 to 0.999 (ablation) | Llama-style 7B · C4 sentences, 7B from scratch | sink ratio: 46.0% -> 20.9% | Hurt |
Closest papers by meaning
- From Attention to Activation: Unraveling the Enigmas of Large Language Models (ICLR 2025 · similarity 0.91)
- What are you sinking? A geometric approach on attention sink (NeurIPS 2025 · similarity 0.89)
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.89)
- Looking Beyond the Top-1: Transformers Determine Top Tokens in Order (ICML 2025 · similarity 0.89)
- Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers (NeurIPS 2026 · similarity 0.89)
- Talking Heads: Understanding Inter-Layer Communication in Transformer Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
