Research map / Architectures and efficiency / Attention and transformers
Theory of self-attention dynamics
Analyzes how self-attention learns, clusters tokens, and relates to causal structure.
43 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2019–2026) · an approach in Attention and transformers, keyword cluster “self attention · attention mechanism · softmax”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
11 papers in 2023–24 and 17 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 1.9% to 1.5%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2019 · Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks (ICML 2019 · 150 citations)
- 2020 · Limits to Depth Efficiencies of Self-Attention (NeurIPS 2020 · 22 citations)
- 2021 · Twins: Revisiting the Design of Spatial Attention in Vision Transformers (NeurIPS 2021 · 612 citations)
- 2022 · Dual Decomposition of Convex Optimization Layers for Consistent Attention in Medical Images (ICML 2022 · 3 citations)
- 2023 · Causal Interpretation of Self-Attention in Pre-Trained Transformers (NeurIPS 2023 · 11 citations)
- 2024 · Learning Correlation Structures for Vision Transformers (CVPR 2024 · 30 citations)
- 2025 · Vision Transformers Don't Need Trained Registers (NeurIPS 2025 · 3 citations)
- 2026 · Kuramoto Oscillatory Phase Encoding: Neuro-inspired Synchronization for Improved Learning Efficiency (ICML 2026 · 0 citations)
Earliest papers here
- Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks (ICML 2019 · 150 citations)
- Area Attention (ICML 2019 · 4 citations)
- Limits to Depth Efficiencies of Self-Attention (NeurIPS 2020 · 22 citations)
What has been tried
30 papers here report 286 measured changes (a component added, removed or swapped, compared with a baseline): helped 63%, no clear effect 11%, hurt 22%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Increase structure dimension from D=0 to D=8 | ImageNet-1K · top-1 accuracy | 80.5 -> 81.3 | Helped | Learning Correlation Structures for Vision Transformers (CVPR 2024 · 30 citations) |
| remove IFA | CelebA-HQ test · ReID rate (ArcFace, cos >0.30) | 0.1 -> 45.5 | Hurt | Facial Identity Anonymization via Intrinsic and Extrinsic Attention Distraction (CVPR 2024 · 16 citations) |
| add skip connections | IMDb · performance | No clear effect | Attention Meets Post-hoc Interpretability: A Mathematical Perspective (ICML 2024 · 4 citations) | |
| zero the 10 register neurons instead of shifting their activations to a register | ImageNet zero-shot classification · Accuracy | 70.4% to 55.6% | Hurt | Vision Transformers Don't Need Trained Registers (NeurIPS 2025 · 3 citations) |
| prune the two lowest-scoring heads | One-shot Omniglot classification (ID) · classification accuracy | 25% | Hurt | Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers (NeurIPS 2024 · 3 citations) |
| Reduce the gradient-descent step size | Noisy training and test data · iterations to benign overfitting | about 150 iterations | Hurt | Benign Overfitting in Single-Head Attention (NeurIPS 2025 · 1 citation) |
| add a feedforward network above the GD attention output | Tiny Stories and Children Stories generation · GPT-4o score | Helped | On Understanding Attention-Based In-Context Learning for Categorical Data (ICML 2025 · 0 citations) | |
| Select high-entropy heads instead of adapting all heads | 11 language modeling benchmarks · average accuracy | 61.9 -> 64.2 | Helped | Understanding Differential Transformer Unchains Pretrained Self-Attentions (NeurIPS 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 6 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet · 2 of 6 papers
- CIFAR-10 · 2 of 6 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 28 papers; from OpenAlex and the papers' affiliations).
- Google · 2 papers
- Zhejiang University · 2 papers
- Massachusetts Institute of Technology · 2 papers
- University of California, Berkeley · 2 papers
Use it
Download BibTeX of 34 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Efficient and robust vision transformers (124 papers)
- Hybrid and interpretable ViT designs (94 papers)
- Token merging and masked pretraining (82 papers)
- Vision transformer pruning (26 papers)
Elsewhere in the topic
- Transformer expressivity and learnability (217 papers)
- State space and recurrent sequence models (172 papers)
- Sparse and structured attention (163 papers)
- Sparse attention for long-context LLMs (147 papers)
- Attention head and block redesign (127 papers)
- Efficient language transformers (105 papers)
- Language model pretraining and induction (99 papers)
- KV cache eviction and compression (92 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).