Research map / Architectures and efficiency / Attention and transformers
How Smooth Is Attention?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies the local Lipschitz constant of unmasked and masked self-attention across regimes (finite sequence length, large radius, mean-field), proving sqrt(n) bounds and a novel mean-field framework for masked attention, validated on BERT and GPT-2.
Problem
The Lipschitz properties of self-attention, key for robustness and expressivity of Transformers, are poorly understood, especially how they depend on sequence length and for masked self-attention.
Approach
Derives Lipschitz bounds for self-attention on compact sets growing as sqrt(n) with sequence length, identifies a large-radius regime where the constant is independent of R, and builds a mean-field framework (with an extra position coordinate) for masked self-attention, showing similar bounds hold.
Key results
- Self-attention Lipschitz constant on compact sets bounded by sqrt(n) up to constant, tight for reasonable sequence lengths.
- On real BERT data the local Lipschitz constant grows roughly as n^(1/4) with sequence length.
- Adversarial inputs built for BERT and GPT-2 yield Lipschitz constants growing like sqrt(n), confirming tightness.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase sequence length on real text inputs | BERT and GPT-2 self-attention · Alice in Wonderland and AG_NEWS | local Lipschitz constant: n^{1/4} | Mixed |
| Increase sequence length on constructed adversarial inputs | Self-attention layers · Adversarial data | local Lipschitz constant: \sqrt{n} | Helped |
| Use pretrained GPT-2 instead of the other real-data cases | GPT-2 self-attention · Real text data | local Lipschitz constant growth rate: faster than \sqrt{n} | Mixed |
| Use smaller scaling factors for adversarial inputs | GPT-2 randomly initialized self-attention · Adversarial data | local Lipschitz constant growth rate: n^{1/2} | Hurt |
Surprising findings
- For pretrained GPT-2, the Lipschitz constant can grow faster than sqrt(n), apparently due to correlation between sequence length and token magnitude after LayerNorm.
Limitations
- Analysis restricted to single-head self-attention; multi-head bound is a loose sum over heads.
- Worst-case bounds may be pessimistic for real data; adversarial construction is heuristic and does not maximize the constant.
- Mean-field bounds grow exponentially in R and are over-pessimistic in practical regimes.
Experimental setup
Datasets: Alice in Wonderland (NLTK Gutenberg) AG_NEWS (torchtext)
Benchmarks: Alice in Wonderland AG_NEWS
Metrics: local Lipschitz constant (operator norm of Jacobian)
Baselines: L2 self-attention Kim et al. mean-field bound
Builds on: L2 self-attention mean-field self-attention (Geshkovski et al.) Wasserstein distance
Closest papers by meaning
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.87)
- Scaling Limits of Long-Context Transformers (NeurIPS 2026 · similarity 0.87)
- Some Fundamental Aspects about Lipschitz Continuity of Neural Networks (ICLR 2024 · similarity 0.86)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.86)
- kNN Attention Demystified: A Theoretical Exploration for Scalable Transformers (ICLR 2025 · similarity 0.86)
- Linear Log-Normal Attention with Unbiased Concentration (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.