Research map / Architectures and efficiency / RNNs and normalization
Understanding and Minimising Outlier Features in Transformer Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies why outlier features (OFs) emerge in transformer training and proposes an Outlier Protected (OP) normalisation-free block plus non-diagonal preconditioners (SOAP) to minimise OFs and improve int8 quantisation.
Problem
Outlier Features (neurons with activation magnitudes far above the layer average) emerge during standard transformer training and hinder low-precision quantisation; little is known about why they emerge or how to minimise them.
Approach
Introduces kurtosis-based OF metrics linked to signal propagation (input-wise Gram matrix), an Outlier Protected transformer block that removes Norm layers while keeping their training benefits (downweighted residuals plus QK-Norm entropy regulation), and shows that non-diagonal (rotated) preconditioners like SOAP and larger diagonal Adam epsilon reduce OFs.
Key results
- The OP block matches Pre-LN perplexity at 100M, 320M and 1.2B scales on Languini Books (e.g. 13.9 vs 13.9 at 1.2B) while reducing kurtosis by up to 4 orders of magnitude.
- On OPT-125m, OP plus SOAP reaches 14.87 W8A8 perplexity (from 14.71 FP16/32), versus 63.4 W8A8 (from 16.00) for the default Pre-LN with AdamW.
- At 7B scale on FineWeb-Edu, Pre-Norm with AdamW shows peak kurtosis over 600, while the OP block stays under 10 and matches Pre-Norm loss.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Pre-LN with the Outlier Protected block | Transformer · Languini Books | Eval PPL: 19.1 -> 18.9; 16.2 -> 16.2; 13.9 -> 13.9 | No clear effect |
| Remove dropout regularisation (ablation) | Pre-LN · OPT-125m, BookCorpus+Wikipedia | W8A8 perplexity: 63.4±50.1 -> 105.9±25.0 | Hurt |
| Increase maximum LR from 4 × 10−4 to 10−3 (ablation) | Pre-LN · OPT-125m, BookCorpus+Wikipedia | W8A8 perplexity: 63.4±50.1 -> 59.0±10.7 | Helped |
| Increase AdamW ϵ from 10−8 to 10−5 (ablation) | Pre-LN · OPT-125m, BookCorpus+Wikipedia | W8A8 perplexity: 63.4±50.1 -> 216.2±87.5 | Hurt |
| Replace AdamW with SOAP (ablation) | Pre-LN · OPT-125m, BookCorpus+Wikipedia | W8A8 perplexity: 63.4±50.1 -> 16.43±0.12 | Helped |
| Remove dropout regularisation (ablation) | Gated Attention [15] · OPT-125m, BookCorpus+Wikipedia | W8A8 perplexity: 16.2±0.09 -> 18.7±0.09 | Hurt |
| Increase maximum LR from 4 × 10−4 to 10−3 (ablation) | Gated Attention [15] · OPT-125m, BookCorpus+Wikipedia | W8A8 perplexity: 16.2±0.09 -> 17.04±0.09 | Hurt |
| Increase AdamW ϵ from 10−8 to 10−5 (ablation) | Gated Attention [15] · OPT-125m, BookCorpus+Wikipedia | W8A8 perplexity: 16.2±0.09 -> 15.54±0.01 | Helped |
| Replace AdamW with SOAP (ablation) | Gated Attention [15] · OPT-125m, BookCorpus+Wikipedia | W8A8 perplexity: 16.2±0.09 -> 15.64±0.01 | Helped |
| Use the Outlier Protected block instead of Pre-LN | Transformer with default optimiser hyperparameters · OPT-125m, BookCorpus+Wikipedia | W8A8 perplexity: 63.4±50.1 -> 16.01±0.01 | Helped |
| Remove dropout regularisation (ablation) | OP · OPT-125m, BookCorpus+Wikipedia | W8A8 perplexity: 16.01±0.01 -> 15.78±0.03 | Helped |
| Increase maximum LR from 4 × 10−4 to 10−3 (ablation) | OP · OPT-125m, BookCorpus+Wikipedia | W8A8 perplexity: 16.01±0.01 -> 15.60±0.01 | Helped |
Surprising findings
- Pre-SRMSNorm (no trainable Norm weights) still shows outlier feature emergence, contradicting the hypothesis that Norm affine parameters cause OFs.
- Non-diagonal preconditioners (SOAP/Shampoo) reduce outlier features and also improve convergence per step, beyond what rotations alone (random rotations in prior work) achieve.
Limitations
- Most scaling experiments at 1.2B and 7B used AdamW only; second-order optimisers were not scaled.
- Limited compute at 7B meant training only 6B tokens and little hyperparameter tuning.
- The link between signal propagation and OFs is argued empirically and not rigorously proven.
Experimental setup
Datasets: CodeParrot Languini Books BookCorpus English Wikipedia FineWeb-Edu ImageNet-style image classification (MLP/ViT)
Benchmarks: Languini Books eval perplexity FineWeb-Edu loss BookCorpus+Wikipedia W8A8 perplexity
Metrics: activation kurtosis training loss perplexity (FP16/32) W8A8 int8 perplexity max-median ratio
Baselines: Pre-LN Post-LN Pre-RMSNorm Pre-SRMSNorm Gated Attention AdamW AdaFactor
Builds on: Pre-Norm transformer Signal Propagation QK-Norm Shampoo AdamW Gated Attention (Bondarenko et al.)
Closest papers by meaning
- Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimization (ICLR 2024 · similarity 0.87)
- Outlier-Efficient Hopfield Layers for Large Transformer-Based Models (ICML 2024 · similarity 0.86)
- Small-scale proxies for large-scale Transformer training instabilities (ICLR 2024 · similarity 0.86)
- Transformers Learn Low Sensitivity Functions: Investigations and Implications (ICLR 2025 · similarity 0.85)
- Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalization (ICLR 2024 · similarity 0.85)
- The Quantization Benefits of Residual-Free Transformers (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.