Research map / Architectures and efficiency / Attention and transformers
Self-attention Networks Localize When QK-eigenspectrum Concentrates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Characterizes attention localization via the eigenspectrum of the query-key matrix (signal propagation probability under a random-walk model), argues small eigenspectrum variance unifies rank and entropy collapse, and proposes the LocAteR regularizer that improves WikiText-2 perplexity in 1-layer transformers.
Problem
Attention localization is linked to two seemingly contradictory failure modes, rank collapse (uniform attention reducing expressivity) and entropy collapse (peaked attention causing plateaus); no unified characterization of when attention localizes and how it affects performance exists.
Approach
Define a signal propagation probability per token (whether a token's signal reaches the QK gradient) under a Gaussian random-walk token model; show localization occurs when the query-key eigenspectrum has a non-zero mean and small variance, which simultaneously mitigates rank collapse and entropy collapse; then regularize the eigenspectrum mean and scale (LocAteR).
Key results
- Under LocAteR with fixed kappa_1=100, stronger mean regularization kappa_2 shrinks the eigenspectrum scale, increases attention entropy, and improves WikiText-2 perplexity for 1-layer, d=128 transformers.
- Signal propagation probability localizes onto few tokens (e.g., middle site) with LocAteR, while remaining uniform without regularization.
- Theoretical lemma: rho localizes when tr(W^2) near zero and |tr(W)| bounded away from zero; uniform when tr(W) near zero with finite tr(W^2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase LocAteR mean regularization strength κ2 while keeping κ1=100 fixed (ablation) | LocAteR, 1-layer Transformer (d=128) · WikiText-2 | perplexity | Helped |
| Increase LocAteR mean regularization strength κ2 while keeping κ1=100 fixed (ablation) | LocAteR, 1-layer Transformer (d=128) · WikiText-2 | attention entropy | Helped |
| Increase LocAteR mean regularization strength κ2 while keeping κ1=100 fixed (ablation) | LocAteR, 1-layer Transformer (d=32) · WikiText-2 | perplexity | Helped |
| Increase LocAteR mean regularization strength κ2 while keeping κ1=100 fixed (ablation) | LocAteR, 3-layer Transformer (d=32) · WikiText-2 | perplexity | Helped |
| Increase LocAteR mean regularization strength κ2 while keeping κ1=100 fixed (ablation) | LocAteR, 6-layer Transformer (d=32) · WikiText-2 | perplexity | Helped |
| Increase LocAteR mean regularization strength κ2 while keeping κ1=100 fixed (ablation) | LocAteR, 3-layer Transformer (d=128) · WikiText-2 | perplexity | Helped |
| Increase LocAteR mean regularization strength κ2 while keeping κ1=100 fixed (ablation) | LocAteR, 6-layer Transformer (d=128) · WikiText-2 | perplexity | Helped |
| Use a smaller QK eigenspectrum scale with a larger eigenspectrum mean | Simulated self-attention · Synthetic random walks (isotropic and anisotropic covariance) | signal propagation localization | Helped |
Surprising findings
- Localization is related to larger eigenspectrum mean and smaller scale, so maximizing the variance-related scale is associated with mitigating both rank and entropy collapse, reconciling the two failure modes.
Limitations
- Relies on a strong Gaussian random-walk token model that does not perfectly match real data.
- Formal analysis restricted to 1-layer transformers.
- Why attention localization improves performance remains only partly understood.
Experimental setup
Datasets: WikiText-2
Benchmarks: WikiText-2
Metrics: perplexity attention entropy eigenspectrum scale tr(W_QK^T W_QK) signal propagation probability
Baselines: No regularization (LocAteR with kappa_2=0)
Builds on: rank collapse analysis of self-attention entropy collapse in transformer training sparsemax
Closest papers by meaning
- Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers (ICML 2025 · similarity 0.88)
- Bayes optimal learning of attention-indexed models (NeurIPS 2025 · similarity 0.87)
- Scaling Limits of Long-Context Transformers (NeurIPS 2026 · similarity 0.87)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.86)
- High-Dimensional Learning Dynamics of Attention-Indexed Models (NeurIPS 2026 · similarity 0.86)
- Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.