Research map / Architectures and efficiency / Attention and transformers
Normalization in Attention Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Casts six transformer normalization schemes (Post-LN, Pre-LN, Mix-LN, Peri-LN, nGPT, sqrt-scaling) as a common normalized attention particle ODE on the sphere, where each scheme is a speed-regulation factor, and derives clustering and initial/terminal velocity results.
Problem
How normalization schemes in deep transformers affect token representation evolution, clustering dynamics, and representation collapse in depth, which has lacked a unified theoretical comparison.
Approach
Reinterprets differently normalized attention updates as a single interacting-particle ODE on the sphere (with Q=K=V=I) where each normalization scheme acts as a speed regulation factor s_j(t); extends Lojasiewicz-based convergence to these modulated gradient flows and compares initial and terminal token velocities across schemes.
Key results
- Pre-LN, Mix-LN and nGPT (constant alpha_t) show polynomial terminal slowdown (C/t^3 for gamma-dot), whereas Post-LN decays as exponential e^{-2t}.
- Peri-LN and nGPT have order-one first-layer displacement, while Post-LN and Pre-LN have initial attention norm bounded by O(sqrt(log n / n) + log n / d).
- Under Pre-LN and Peri-LN with n <= e^beta, tokens synchronize unconditionally since the radial velocity is bounded below by 1/(n e^beta).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Peri-LN instead of Post-LN | Normalized attention dynamics · Symmetric orthogonal initialization | terminal cosine-similarity velocity: Ce^{-2t} -> C/t^3 | Helped |
| Use Pre-LN instead of Post-LN | Normalized attention dynamics · Symmetric orthogonal initialization | terminal cosine-similarity velocity: Ce^{-2t} -> C/t^3 | Helped |
| Use sqrt-scaling instead of Post-LN | Normalized attention dynamics · Symmetric orthogonal initialization | terminal cosine-similarity velocity: Ce^{-2t} -> \displaystyle C\frac{e^{-4\sqrt{t}}}{\sqrt{t}} | Helped |
| Use Peri-LN instead of Post-LN | Normalized attention dynamics · Local-cone initialization | variance decay rate: -\Theta(\mathrm{Var}(t)) -> -\Theta\left(\mathrm{Var}(t)/t \right) | Helped |
| Use Pre-LN instead of Post-LN | Normalized attention dynamics · Local-cone initialization | variance decay rate: -\Theta(\mathrm{Var}(t)) -> -\Theta\left(\mathrm{Var}(t)/t \right) | Helped |
| Use Peri-LN instead of Post-LN | Normalized attention dynamics · Random uniform initial directions | first-layer angular displacement: \Omega(\min(d/\log n, \sqrt{n/\log n})) | Helped |
| Use nGPT with a selected \alpha_t schedule | Normalized attention dynamics · Symmetric orthogonal initialization | terminal cosine-similarity velocity: C\displaystyle\alpha_te^{-2\int_C^t \alpha_s d s} | Mixed |
Surprising findings
- Pre-LN and Mix-LN, often viewed as more stable, exhibit polynomial rather than exponential clustering, so they keep tokens distinct across many layers.
Limitations
- Analysis omits MLP layers and assumes tied/restricted Q, K, V matrices (Q=K=V=I or V=Q^T K).
- Only initial and terminal velocities are bounded; intermediate regime and metastability are not characterized.
- No experiments with trained transformers; predictions (Pre-LN norm growth linear vs sqrt(t) empirically) differ from practice.
Experimental setup
Metrics: cosine similarity evolution initial velocity bound cluster variance decay rate
Baselines: Post-LN Pre-LN Mix-LN Peri-LN nGPT sqrt-scaling
Builds on: Lojasiewicz inequality particle dynamics of transformers (Geshkovski et al.) nGPT Mix-LN Peri-LN
Closest papers by meaning
- Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers (NeurIPS 2026 · similarity 0.88)
- Looking Beyond the Top-1: Transformers Determine Top Tokens in Order (ICML 2025 · similarity 0.86)
- JoMA: Demystifying Multilayer Transformers via Joint Dynamics of MLP and Attention (ICLR 2024 · similarity 0.86)
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.86)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.85)
- Peri-LN: Revisiting Normalization Layer in the Transformer Architecture (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.