Research map / Architectures and efficiency / Attention and transformers
Subcritical Signal Propagation at Initialization in Normalization-Free Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends the averaged partial Jacobian norm (APJN) signal-propagation theory to bidirectional-attention transformers under permutation-symmetric tokens, showing tanh-like normalization-free (DyT/Derf) transformers are subcritical with stretched-exponential APJN growth, and validates with ViT experiments on CIFAR-100.
Problem
Signal propagation and gradient amplification at initialization in transformers that replace LayerNorm with elementwise tanh-like nonlinearities (DyT/Derf), and how this affects trainability and sensitivity to hyperparameters.
Approach
Derives recurrences for activation covariances and APJNs in transformers with uniform bidirectional attention under permutation-symmetric token configurations and a Gaussian mean-field approximation, yielding power-law APJN growth for pre-LN and stretched-exponential (subcritical) growth for tanh-like normalization-free transformers.
Key results
- On synthetic permutation-symmetric inputs, theory accurately predicts backward APJNs in pre-LN and Derf ViTs.
- On CIFAR-100 with a 128-block ViT, theoretical backward-APJN curves achieve per-sample GMFE typically below 1.25 in middle and deep blocks.
- Derf trainability on CIFAR-100 is more sensitive to depth and weight initialization than pre-LN; smaller alpha improves stability but alpha=0.1 slows convergence.
Surprising findings
- An overly small initial alpha (e.g., 0.1) in Derf can substantially slow convergence, so decreasing alpha is not free.
Limitations
- Experiments limited to ViT on CIFAR-100 with truncated training runs (10-20 epochs), so long-term dynamics and final performance are not assessed.
- Theory relies on simplifying assumptions (uniform attention, permutation-symmetric tokens, Gaussian mean-field), with extended theory needed for larger sigma_OV.
- Training-stability sweeps are single-run experiments, interpreted qualitatively.
Experimental setup
Datasets: CIFAR-100 synthetic permutation-symmetric token inputs
Benchmarks: CIFAR-100
Metrics: backward APJN geometric mean fold error test accuracy
Baselines: pre-LN ViT
Builds on: APJN framework for residual networks Dynamic Tanh Derf pre-LayerNorm
Closest papers by meaning
- Unraveling the Gradient Descent Dynamics of Transformers (NeurIPS 2024 · similarity 0.88)
- A unified framework for establishing the universal approximation of transformer-type architectures (NeurIPS 2025 · similarity 0.87)
- Peri-LN: Revisiting Normalization Layer in the Transformer Architecture (ICML 2025 · similarity 0.87)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.87)
- Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers (NeurIPS 2026 · similarity 0.86)
- Simplifying Transformer Blocks (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.