Research map / Architectures and efficiency / Attention and transformers
Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
Enrique Queipo-de-Llano, Alvaro Arroyo, Federico Barbero, Xiaowen Dong, Michael M. Bronstein, Yann LeCun, Ravid Shwartz-Ziv
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| zero the MLP contribution to the BOS token at layer 0 (removes massive activation) (ablation) | LLaMA3 8B · GSM8K | matrix-based entropy (bits): 0.02 -> 0.5 | Improved |
| zero the MLP contribution to the BOS token at layer 0 (removes massive activation) (ablation) | LLaMA3 8B · GSM8K | sink rate: 0.85 -> 0.0 | Hurt |
| zero the MLP contribution to the BOS token at layer 0 (removes massive activation) (ablation) | LLaMA3 8B · GSM8K | BOS norm ratio to other tokens: 10^2x -> within 2x | Hurt |
| ablate only a single massive-activation stage (multi-stage models) (ablation) | Pythia 410M, Qwen2 7B · GSM8K | compression: partially reduces compression | Mixed |
| clip magnitude of the massive activation to lower norm (ablation) | Pythia 410M · GSM8K | representational entropy and sink rate: entropy rises, sinks decrease with general downward trend | Mixed |
| go from early layers to full depth via LogitLens | pretrained LLMs · WikiText-2 | perplexity: >10^4 -> 10-25 | Improved |
| read out mid-depth layers (25-75% depth) instead of early/late layers for embeddings | decoder-only LLMs · MTEB (32 tasks) | performance: 10-20% | Improved |
Closest papers by meaning
- When Attention Sink Emerges in Language Models: An Empirical View (ICLR 2025 · similarity 0.89)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.88)
- When Less is More: The LLM Scaling Paradox in Context Compression (NeurIPS 2026 · similarity 0.88)
- Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs (AAAI 2025 · similarity 0.88)
- Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression (ICML 2025 · similarity 0.88)
- On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
