Research map / Architectures and efficiency / Attention and transformers
Contribution Weights: A Geometrical Analysis of Self-Attention Transformers
Harry Jake Cunningham, Nicola Muca Cirone
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cosine alignment to attention weights as predictor of contribution | regression of contribution weight, tail tokens · FineWeb-Edu, j>1 | R^2: 0.449 -> 0.834 | Improved |
| add relative norm to attention weights as predictor | regression of contribution weight, tail tokens · FineWeb-Edu, j>1 | R^2: 0.449 -> 0.579 | Improved |
| use relative norm and cosine instead of attention alone for initial token | regression of contribution weight, initial token · FineWeb-Edu, j=1 | R^2: 0.079 -> 0.725 | Improved |
| scale sink value-vector norm down from 1 toward 0 (ablation) | Llama-3.1-8B · FineWeb-Edu 128 sequences | perplexity: perplexity increases monotonically as gamma decreases | Hurt |
| zero the sink value-vector norm (ablation) | Llama-3.1-8B small-norm head outputs · semantic cone analysis | head outputs inside semantic cone: small-norm head outputs lie within the semantic cone | Hurt |
| add value-norm weighting to attention (VW) as token importance measure | attention-based token importance · GSM8K / FineWeb-Edu token removal | comprehensiveness/sufficiency: performance improves when attention weights multiplied by value norms | Improved |
| add directional alignment (contribution weights) over VW-ON | token importance measure · token removal, 4 LLMs | Delta PPL: largest performance increase among all metrics | Improved |
| allow sink token removal (ablation) | attention-based importance · token removal | language modeling loss: attention-based metrics induce immediate model collapse | Hurt |
| use global percentiles instead of layer-specific thresholds (ablation) | contribution weights · token removal | language modeling loss: gap narrows compared to layer-specific normalization | Mixed |
| replace attention by contribution weights for first-token weight | LLaMA-8B · FineWeb-Edu | average weight on first token: 0.708 -> 0.142 | Improved |
| replace attention by contribution weights for tail-token entropy | LLaMA-8B · FineWeb-Edu j!=1 | entropy: 2.61 -> 2.27 | Improved |
| evaluate on FineWeb-Edu instead of GSM8K (ablation) | contribution weights · token removal | Delta PPL: hierarchy of improvement remains consistent | No clear effect |
Closest papers by meaning
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.89)
- Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings (NeurIPS 2025 · similarity 0.89)
- On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs (NeurIPS 2026 · similarity 0.89)
- When Attention Sink Emerges in Language Models: An Empirical View (ICLR 2025 · similarity 0.89)
- Dual Dimensionality for Local and Global Attention (NeurIPS 2026 · similarity 0.88)
- Attention with Trained Embeddings Provably Selects Important Tokens (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
