Research map / Architectures and efficiency / Attention and transformers
Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses random matrix theory to show softmax attention at initialisation has a spectral gap that causes rank collapse in width (context length), and proposes removing the outlier eigenvalue (A minus 1/T all-ones) to mitigate rank collapse and gradient explosion.
Problem
Signal and gradient propagation through randomly initialised softmax attention layers is pathological: vanishing/exploding gradients and rank collapse in depth, and a newly identified rank collapse in width (growing context length) that is unique to attention.
Approach
Softmax attention matrices with isotropic inputs are i.i.d. Markov matrices whose top eigenvalue escapes to 1 forming a spectral gap; subtracting the rank-one mean component (1/T)·1 from the attention matrix removes the outlier, preventing stable-rank collapse in width and mitigating depth collapse and gradient explosion.
Key results
- Theorem shows the stable rank of the covariance of a single softmax attention layer converges to 1 as context length T grows, at rate O(T^-3), under isotropic inputs.
- Removing the rank-one outlier (A minus (1/T)·ones) yields stable rank scaling linearly in T, and gradient norms stay flat with depth rather than growing exponentially.
- Removing all outliers in deep trained-style attention is found to be nearly as effective as removing only the dominant one, at much lower cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Subtract the uniform rank-one component from softmax attention | Single attention layer · Isotropic input; increasing context length | stable rank: 1 -> scales linearly in width | Helped |
| Subtract the uniform rank-one component at every layer | Deep i.i.d. Markov attention network · Increasing context length | stable rank: 1 -> scales linearly in width | Helped |
| Subtract the uniform rank-one component at every layer | Deep i.i.d. Markov attention network · Increasing depth | squared gradient norm: T^{L-1} -> grow linearly with d | Helped |
| Add LayerNorm to the attention block (ablation) | Transformer at initialization · Increasing context length | stable rank | No clear effect |
| Add skip connections to the attention block (ablation) | Transformer at initialization · Increasing context length | stable rank | No clear effect |
| Add LayerNorm and skip connections to the attention block (ablation) | Transformer at initialization · Increasing context length | stable rank | No clear effect |
| Replace the attention matrix with the identity matrix (ablation) | Transformer at initialization · Increasing depth | stable rank | Hurt |
| Use orthogonal value matrices (ablation) | Transformer at initialization · Increasing depth | stable rank | Helped |
| Add LayerNorm or remove the spectral gap | i.i.d. Markov attention network · Increasing depth | gradient norm: exponentially exploding -> flat | Helped |
| Remove all spectral outliers rather than only the largest (ablation) | Deep transformers · Rank collapse and gradient norms | rank-collapse mitigation: comparable results | No clear effect |
| Increase the context-to-embedding ratio beyond 1 (ablation) | TinyBert and synthetic inputs · γ > 1 | stable rank | No clear effect |
| Use random token embeddings instead of pretrained-tokenizer embeddings (ablation) | Randomly initialized RoBERTa encoder · Non-isometric input tokens | spectral gap | No clear effect |
Surprising findings
- Additional outlier eigenvalues emerge in the attention spectrum of deeper layers in practical models.
- Rank collapse in width appears in pretrained-architecture BERT encoders at initialisation, and the spectral gap persists even when context length exceeds embedding dimension.
Limitations
- Theory assumes isotropic inputs (orthonormal rows) and i.i.d. Gaussian weights; the paper says these are for mathematical rigour.
- Analysis concerns initialisation only; training dynamics are explicitly out of scope.
- Asymptotic results rely on large-width limits and freeness assumptions.
Experimental setup
Datasets: sentences from the paper abstract (tokenised inputs) synthetic isotropic Gaussian inputs
Metrics: stable rank eigenvalue spectrum gradient norm
Baselines: vanilla softmax attention LayerNorm skip connections Differential Transformer
Builds on: Differential Transformer rank collapse analysis (Dong et al.) free probability i.i.d. Markov matrix spectral results
Closest papers by meaning
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.92)
- Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programs (NeurIPS 2025 · similarity 0.89)
- Self-attention Networks Localize When QK-eigenspectrum Concentrates (ICML 2024 · similarity 0.88)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.88)
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.88)
- Scaling Limits of Long-Context Transformers (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.