AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Contribution Weights: A Geometrical Analysis of Self-Attention Transformers

Harry Jake Cunningham, Nicola Muca Cirone

ICML 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse and structured attention · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add cosine alignment to attention weights as predictor of contributionregression of contribution weight, tail tokens · FineWeb-Edu, j>1R^2: 0.449 -> 0.834Improved
add relative norm to attention weights as predictorregression of contribution weight, tail tokens · FineWeb-Edu, j>1R^2: 0.449 -> 0.579Improved
use relative norm and cosine instead of attention alone for initial tokenregression of contribution weight, initial token · FineWeb-Edu, j=1R^2: 0.079 -> 0.725Improved
scale sink value-vector norm down from 1 toward 0 (ablation)Llama-3.1-8B · FineWeb-Edu 128 sequencesperplexity: perplexity increases monotonically as gamma decreasesHurt
zero the sink value-vector norm (ablation)Llama-3.1-8B small-norm head outputs · semantic cone analysishead outputs inside semantic cone: small-norm head outputs lie within the semantic coneHurt
add value-norm weighting to attention (VW) as token importance measureattention-based token importance · GSM8K / FineWeb-Edu token removalcomprehensiveness/sufficiency: performance improves when attention weights multiplied by value normsImproved
add directional alignment (contribution weights) over VW-ONtoken importance measure · token removal, 4 LLMsDelta PPL: largest performance increase among all metricsImproved
allow sink token removal (ablation)attention-based importance · token removallanguage modeling loss: attention-based metrics induce immediate model collapseHurt
use global percentiles instead of layer-specific thresholds (ablation)contribution weights · token removallanguage modeling loss: gap narrows compared to layer-specific normalizationMixed
replace attention by contribution weights for first-token weightLLaMA-8B · FineWeb-Eduaverage weight on first token: 0.708 -> 0.142Improved
replace attention by contribution weights for tail-token entropyLLaMA-8B · FineWeb-Edu j!=1entropy: 2.61 -> 2.27Improved
evaluate on FineWeb-Edu instead of GSM8K (ablation)contribution weights · token removalDelta PPL: hierarchy of improvement remains consistentNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy