Research map / Architectures and efficiency / RNNs and normalization
Learning in Compact Spaces with Approximately Normalized Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes anGPT, a transformer that replaces normalization layers with input-independent scalar normalization factors (justified by norm concentration in high dimensions) and bounded weights, achieving about 1.4x faster convergence than GPT+ with QK norm at ~3% runtime overhead.
Problem
Normalization stabilizes deep transformer training and speeds convergence, but exact normalization layers add training and inference overhead and require extra hyperparameters (weight decay, warm-up); residual-stream norm growth ("curse of depth") also destabilizes training.
Approach
Replace exact normalization with constant scalar normalization factors derived from the concentration of norms of high-dimensional random vectors, bound (rather than normalize) weight norms, use LERP residual updates with a norm factor nu(alpha)=1-2alpha+2alpha^2, and decouple stabilization from learnable gains so no weight decay or warm-up is needed.
Key results
- anGPT achieves an average 1.4x convergence speedup over GPT+ (QK norm) on 0.5B models across 5B-70B token budgets, and 2.0x over GPT+ without QK norm.
- Training runtime per step increases by about 2.75% for anGPT versus 9.6% for nGPT.
- On 1B models, anGPT outperforms GPT+ on six downstream benchmarks across three budgets, with gains of 3% to 22%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use anGPT instead of GPT+ | 1B GPT · PIQA, 42B Tokens | accuracy: 0.700 -> 0.718 | Helped |
| use anGPT instead of GPT+ | 1B GPT · ARC-Easy, 42B Tokens | accuracy: 0.561 -> 0.596 | Helped |
| use anGPT instead of GPT+ | 1B GPT · HellaSwag, 42B Tokens | accuracy: 0.400 -> 0.413 | Helped |
| use anGPT instead of GPT+ | 1B GPT · LAMBADA, 42B Tokens | perplexity: 16.726 -> 14.679 | Helped |
| use anGPT instead of GPT+ | 1B GPT · WikiText, 42B Tokens | perplexity: 17.621 -> 16.175 | Helped |
| use anGPT instead of GPT+ | 1B GPT · WinoGrande, 42B Tokens | accuracy: 0.554 -> 0.563 | Helped |
| use anGPT instead of GPT+ | 1B GPT · MMLU, 42B Tokens | accuracy: 0.243 -> 0.256 | Helped |
| use anGPT instead of GPT+ | 0.5B GPT · SlimPajama, 5B–70B tokens | convergence speedup: 1.4× | Helped |
| replace GPT+ with nGPT | 0.5B GPT · SlimPajama, 5B–70B tokens | convergence speedup: 1.29× | Helped |
| replace GPT+ with anGPT | 0.5B model training step · Runtime comparison | average runtime per step: 0.1416 -> 0.1455 | Hurt |
| scale one constant normalization factor by 0.5 (ablation) | anGPT · OpenWebText, 0.5B, 10B tokens | validation loss: 2.638 -> 2.647 | Hurt |
| scale one constant normalization factor by 2.0 (ablation) | anGPT · OpenWebText, 0.5B, 10B tokens | validation loss: 2.638 -> 2.642 | Hurt |
Surprising findings
- Adding QK norm to the GPT baseline yields a notable gain, and the authors find their GPT baseline results differ from reported nGPT speedups of 4x to 20x.
- The derivation's assumptions (uniform sphere distribution, independence) are violated by design via bounded weights, yet norm concentration is still observed empirically.
Limitations
- No experiments beyond 7x Chinchilla optimal token budget, so performance at larger budgets is unknown.
- Single runs without error bars or multiple seeds; experiments mostly on one dataset and only the GPT architecture.
- Attention matrix normalization is not included in the architecture because sparse attention (e.g., causal/FlashAttention) makes the factors inaccurate.
Experimental setup
Datasets: SlimPajama OpenWebText
Benchmarks: SlimPajama validation loss OpenWebText validation loss PIQA ARC-Easy HellaSwag LAMBADA WikiText WinoGrande MMLU
Metrics: validation loss perplexity convergence speedup accuracy runtime per step
Baselines: GPT+ nGPT GPT without QK norm DyT LN scaling
Builds on: nGPT QK normalization Curse of Depth Chinchilla scaling laws RMSNorm
Closest papers by meaning
- DNT: a Deeply Normalized Transformer that can be trained by Momentum SGD (ICLR 2026 · similarity 0.90)
- nGPT: Normalized Transformer with Representation Learning on the Hypersphere (ICLR 2025 · similarity 0.87)
- HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization (NeurIPS 2025 · similarity 0.87)
- Transformers without Normalization (CVPR 2025 · similarity 0.86)
- Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion (ICLR 2024 · similarity 0.85)
- Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unity (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.