Research map / Architectures and efficiency / Attention and transformers
Geometry of Lightning Self-Attention: Identifiability and Dimension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies the geometry of lightning (unnormalized) self-attention networks via algebraic geometry, characterizing parameter fibers and computing neuromanifold dimension for single and deep layers, plus a conjecture for softmax attention verified numerically.
Problem
Understanding the geometry (identifiability, fibers, dimension, singularities) of the function spaces (neuromanifolds) defined by self-attention networks, which are not studied for attention despite their importance for sample complexity and training dynamics.
Approach
Treats unnormalized (lightning) self-attention as polynomial (trilinear in weights, cubic in input), reparametrizes deep networks via virtual weights (M, L), and uses unique factorization and induction to show generic fibers are only rescalings and symmetries, yielding closed-form dimension; conjectures and numerically verifies the analogous result for softmax attention.
Key results
- For a single lightning attention layer with t>=2, the neuromanifold dimension is 2ad + dd' - a^2 - 1 when a <= d (generic fibers are one-dimensional up to rescaling).
- For deep bottleneck lightning attention networks, the generic fiber consists only of rescalings, giving an explicit dimension formula 2 alpha_1 d_0 - alpha_1^2 + delta(d_0+d_l) - delta^2 - l + sum_{i>1}(2 alpha_i delta - alpha_i^2).
- For a single softmax attention layer the parametrization is generically one-to-one; numerical Jacobian rank for a 2-layer softmax network matches the conjectured dimension for delta from 3 to 10.
Surprising findings
- Singularities of the single-layer neuromanifold occur exactly when rank(A)·rank(V) <= 1, suggesting an implicit bias toward low-rank functions.
Limitations
- Analysis restricted to lightning (unnormalized, homogeneous) attention without skip connections, multiple heads, or biases.
- Softmax deep-case result is only a conjecture supported by numerical experiments for one small architecture (l=2, t=3, a=2).
- Generic-fiber results require assumptions such as bottleneck dimensions d_i equal and rank/nonskew-symmetry conditions.
Experimental setup
Metrics: dimension of neuromanifold (Jacobian rank)
Builds on: Lightning self-attention determinantal variety Singular Learning Theory
Closest papers by meaning
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.85)
- Attention Mechanism, Max-Affine Partition, and Universal Approximation (NeurIPS 2025 · similarity 0.85)
- Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers (ICML 2025 · similarity 0.84)
- Clustering in Causal Attention Masking (NeurIPS 2024 · similarity 0.84)
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.83)
- Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.