Research map / Architectures and efficiency / Attention and transformers
nGPT: Normalized Transformer with Representation Learning on the Hypersphere
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes nGPT, a Transformer whose embeddings, attention and MLP matrices and hidden states are all unit-norm on a hypersphere with learnable eigen learning rates, reaching the same validation loss 4-20x faster than GPT on OpenWebText.
Problem
Standard Transformers have unconstrained weight and hidden-state norms, which make training slow and ill-conditioned (e.g., high condition numbers in attention matrices) and rely on weight decay and normalization layers.
Approach
Normalize all embedding-dimension vectors of matrices, embeddings and hidden states to the unit hypersphere, treat each attention/MLP block output as a gradient-like direction, and update the hidden state via learnable per-dimension eigen learning rates (LERP on the sphere with renormalization as Riemannian retraction), removing weight decay and warmup.
Key results
- nGPT reaches in 20k iterations the validation loss GPT reaches only after 200k iterations for 1B models at 4k context (about 10x speedup).
- Training of 0.5B and 1B nGPT is about 4x, 10x and 20x faster in tokens at 1k, 4k and 8k context lengths.
- Attention matrices of GPT have significantly higher condition numbers than nGPT; the learned eigen learning rates average about 0.20-0.25 for attention and 0.32-0.37 for MLP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GPT with the normalized Transformer (nGPT) | GPT · OpenWebText, 1k context | training speedup (tokens): 4x | Helped |
| replace GPT with the normalized Transformer (nGPT) | GPT · OpenWebText, 4k context | training speedup (tokens): 10x | Helped |
| replace GPT with the normalized Transformer (nGPT) | GPT · OpenWebText, 8k context | training speedup (tokens): 20x | Helped |
| replace GPT with the normalized Transformer (nGPT) | GPT · OpenWebText, 1B model, 4k context | training speedup (iterations): 20k iterations vs 200k iterations | Helped |
| use fixed values for s_qk, s_u, and s_v and a single global learnable s_z (ablation) | nGPT · OpenWebText downstream tasks | accuracy: ≤0.3% in validation loss | Hurt |
| remove QK normalization (ablation) | nGPT · in-distribution downstream tasks | accuracy | No clear effect |
| remove QK normalization (ablation) | nGPT · length extrapolation | perplexity | Hurt |
| remove QK normalization (ablation) | nGPT · QK normalization computation | computational savings: about 12% | Helped |
| use unconstrained rather than positive eigen learning rates (ablation) | nGPT · experimental results | performance | No clear effect |
| simplify per-element scaling factors or eigen learning rates to a learned scalar or fixed value (ablation) | nGPT · validation loss | validation loss: ≤0.3% | Mixed |
Surprising findings
- Making eigen learning rates unconstrained (allowing negative values) yields essentially the same results, since sign flips can be absorbed into the output projection matrix.
- Removing QK-normalization gives nearly equal in-distribution performance but worse length extrapolation.
Limitations
- Wall-clock time per step is higher than GPT (about 60-80% overhead) due to unfused normalization.
- Experiments limited to 0.5B and 1B models on OpenWebText; scaling to larger models and real-world datasets is future work.
- Some 1B 8k-context runs showed increased sensitivity to hyperparameters.
Experimental setup
Datasets: OpenWebText PG19
Benchmarks: downstream task suite (average accuracy) PG19 perplexity WMT14 FR-EN translation (BLEU)
Metrics: validation loss downstream accuracy perplexity BLEU condition number
Baselines: GPT (Transformer with RMSNorm, weight decay) ReZero NormFormer
Builds on: Transformer RoPE SwiGLU Adam QK-normalization
Closest papers by meaning
- Learning in Compact Spaces with Approximately Normalized Transformer (NeurIPS 2025 · similarity 0.87)
- Normalized Architectures are Natively 4-Bit (NeurIPS 2026 · similarity 0.86)
- Transformers without Normalization (CVPR 2025 · similarity 0.84)
- HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization (NeurIPS 2025 · similarity 0.84)
- Normalization in Attention Dynamics (NeurIPS 2025 · similarity 0.84)
- SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.