Research map / Architectures and efficiency / Attention and transformers
From Attention to Activation: Unraveling the Enigmas of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies first-token attention dominance and outlier activations in autoregressive Transformers, proposes softmax-1 to remove attention sinks on the first token and OrthoAdam (orthogonally rotated Adam) to suppress outlier hidden-state activations, improving robustness to basic quantisation.
Problem
Autoregressive Transformers show two anomalies: the first token dominates attention in most heads, and large outlier activations appear in specific hidden-state channels; these hurt quantisation and long-context extension.
Approach
Attribute first-token attention dominance to the softmax normalisation constraint (fixed by adding 1 to the denominator so heads can attend nowhere) and outlier activations to adaptive optimisers tracking moments in the parameter basis (fixed by applying random orthogonal rotations to gradients before Adam moment updates).
Key results
- softmax-1 plus OrthoAdam cuts GPT2-1.4B mean layer kurtosis from 351.0 to 3.0 with perplexity 13.3 vs 13.4 for baseline.
- Vanilla GPT2-1.4B under 4-bit zeropoint weight quantisation reaches perplexity 3577.7; with softmax-1 and OrthoAdam it is 13.6.
- First-token maximum attention drops from up to 64.8% to 1-3% with softmax-1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace canonical softmax with softmax-1 | GPT2 attention · C4 en validation | first-token attention proportion: 0.527 -> 0.024 | Helped |
| Use OrthoAdam instead of Adam | GPT2 with softmax-1 · C4 en validation, GPT2 130M | mean kurtosis, tokens >1: 144.2 -> 7.3 | Helped |
| Use softmax-1 and OrthoAdam | GPT2 350M · C4 en validation | PPL: 16.4 -> 16.3 | No clear effect |
| Use OrthoAdam and softmax-1 | GPT2 1.4B · C4 en validation, 4-bit Zeropoint weight quantization | PPL: 3577.7 -> 13.6 | Helped |
| Remove biases from linear layers (ablation) | GPT2 with canonical softmax and Adam · C4 en validation, GPT2 130M ablation | first-token attention proportion: 0.333 -> 0.308 | No clear effect |
| Remove positional encodings (ablation) | GPT2 with canonical softmax and Adam · C4 en validation, GPT2 130M ablation | first-token attention proportion: 0.333 -> 0.197 | Mixed |
| Replace absolute positional encoding with rotary encoding (ablation) | GPT2 with canonical softmax and Adam · C4 en validation, GPT2 130M ablation | PPL: 26.9 -> 25.6 | Helped |
| Replace absolute positional encoding with no positional encoding (ablation) | GPT2 with canonical softmax and Adam · C4 en validation, GPT2 130M ablation | PPL: 26.9 -> 27.4 | Hurt |
| Replace LayerNorm with RMSNorm-M (ablation) | GPT2 with softmax-1 and Adam · C4 en validation, GPT2 130M ablation | mean kurtosis: 244.7 -> 230.4 | No clear effect |
| Replace LayerNorm with RMSNorm-S (ablation) | GPT2 with softmax-1 and Adam · C4 en validation, GPT2 130M ablation | mean kurtosis: 244.7 -> 140.0 | Helped |
| Replace Adam with RMSProp (ablation) | GPT2 with softmax-1 and RMSNorm-S · C4 en validation, GPT2 130M ablation | mean kurtosis: 140.0 -> 70.5 | Helped |
| Replace Adam with RMSProp (ablation) | GPT2 with softmax-1 and RMSNorm-S · C4 en validation, GPT2 130M ablation | PPL: 26.9 -> 27.4 | Hurt |
Surprising findings
- Outlier activations persist even after removing first-token attention dominance with softmax-1, suggesting the two phenomena are separate.
- Adam and RMSProp produce high kurtosis (140 and 70), while SGD yields kurtosis near 3, implicating adaptive moment tracking in the parameter basis.
Limitations
- Main GPT2-1.4B model trained for fewer steps (600k) due to computational constraints.
- Experiments are limited to small models trained from scratch on C4; pretrained large-model fixes are not tested.
- Some ablations (e.g., GPT2-60M/130M final-layer kurtosis rises) are left unexplained.
Experimental setup
Datasets: C4 en
Benchmarks: C4 en validation
Metrics: perplexity activation kurtosis max absolute activation first-token attention percentage quantised perplexity
Baselines: canonical softmax Adam RMSProp SGD LayerNorm LLM.int8 SmoothQuant
Builds on: Adam GPT-2 Llama 2 RMSNorm
Closest papers by meaning
- The Closeness of In-Context Learning and Weight Shifting for Softmax Regression (NeurIPS 2024 · similarity 0.89)
- Systematic Outliers in Large Language Models (ICLR 2025 · similarity 0.89)
- Sparsing Law: Towards Large Language Models with Greater Activation Sparsity (ICML 2025 · similarity 0.89)
- When Attention Sink Emerges in Language Models: An Empirical View (ICLR 2025 · similarity 0.89)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.88)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.