Research map / Architectures and efficiency / Attention and transformers
What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives the exact Hessian of a single self-attention layer (classical and linear variants) in matrix calculus, characterizes its data, attention-moment and weight dependencies, and contrasts it with MLP/CNN Hessians.
Problem
Transformers differ from MLPs and CNNs in ways that require adaptive optimizers, layer norm and warmup, but the root causes in the loss landscape are poorly understood; no exact theoretical Hessian of self-attention existed.
Approach
Derives the full Gauss-Newton-decomposed Hessian of a single self-attention layer via matrix derivatives, expresses it via attention central moment matrices, and shows the query/key/value blocks have heterogeneous, highly nonlinear dependence on data and weights, with softmax and query-key parameterization driving this heterogeneity.
Key results
- The value diagonal Hessian block scales as sigma^2 and the query block follows the sigma^5 functional-Hessian trend, confirming block heterogeneity in data dependence.
- Query-key outer-product Hessian has cubic data dependence per weight, while value has linear dependence (O(X^2) for value block of outer product Hessian).
- Removing softmax makes block entry magnitudes more homogeneous and the diagonal value functional Hessian is zero, so diagonal blocks become PSD without softmax.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove softmax activation (ablation) | Self-attention · GPT-2 single-block next-token prediction at initialization | Hessian block-entry magnitude heterogeneity | Helped |
| Remove softmax activation (ablation) | Self-attention · GPT-2 single-block next-token prediction at initialization | Query-versus-value Hessian block magnitude: two orders of magnitude smaller | Helped |
| Remove softmax activation (ablation) | Self-attention · Self-attention Hessian | Diagonal-block positive semidefiniteness | Helped |
| Use Pre-LN (ablation) | Transformer block · Hessian block Frobenius norm across input scales | Inter-block data heterogeneity | Helped |
| Increase embedding scale | GPT-2 single-block digit addition · Digit addition | Value-block Hessian Frobenius norm: σ^2 | Mixed |
| Increase embedding scale | GPT-2 single-block digit addition · Digit addition | Query-block Hessian Frobenius norm: σ^5 | Mixed |
Surprising findings
- Softmax, not only the architecture's linearity, drives the Hessian's indefiniteness in diagonal blocks, because without softmax the functional Hessian is block-hollow.
- For a D-layer linear self-attention, the Hessian contains 3^D intra-sequence covariance factors, unlike deep linear MLPs.
Limitations
- Theory restricted to a single self-attention layer with MSE loss; extension to deep Transformers is left open.
- Embedding matrix is treated as fixed rather than learned.
- Empirical validation uses a single GPT-2 block on a digit addition task.
Experimental setup
Datasets: digit addition task (next-token prediction)
Metrics: Frobenius norm of Hessian diagonal blocks growth exponent in sigma
Baselines: MLP Hessian CNN Hessian linear self-attention single-matrix query-key parameterization
Builds on: Gauss-Newton decomposition Hessian of deep linear networks Kronecker calculus
Closest papers by meaning
- Unraveling the Gradient Descent Dynamics of Transformers (NeurIPS 2024 · similarity 0.89)
- Viewing Transformers Through the Lens of Long Convolutions Layers (ICML 2024 · similarity 0.88)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.88)
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.87)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.87)
- A unified framework for establishing the universal approximation of transformer-type architectures (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.