Research map / Architectures and efficiency / Attention and transformers
Transformer expressivity and learnability
Theoretically analyzes what algorithms and structured data transformers can represent and learn.
217 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2019–2026) · an approach in Attention and transformers, keyword cluster “vision · vit · transformers”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
65 papers in 2023–24 and 134 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 11.1% to 11.5%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2019 · Novel positional encodings to enable tree-based transformers (NeurIPS 2019 · 83 citations)
- 2020 · Are Transformers universal approximators of sequence-to-sequence functions? (ICLR 2020 · 40 citations)
- 2021 · Relative Positional Encoding for Transformers with Linear Complexity (ICML 2021 · 20 citations)
- 2022 · Frozen Pretrained Transformers as Universal Computation Engines (AAAI 2022 · 111 citations)
- 2023 · Graphix-T5: Mixing Pre-trained Transformers with Graph-Aware Layers for Text-to-SQL Parsing (AAAI 2023 · 88 citations)
- 2024 · Permutation Equivariance of Transformers and Its Applications (CVPR 2024 · 16 citations)
- 2025 · Transformer Layers as Painters (AAAI 2025 · 10 citations)
- 2026 · Parameterized Abstract Interpretation for Transformer Verification (AAAI 2026 · 2 citations)
Earliest papers here
- Novel positional encodings to enable tree-based transformers (NeurIPS 2019 · 83 citations)
- Are Transformers universal approximators of sequence-to-sequence functions? (ICLR 2020 · 40 citations)
- Robustness Verification for Transformers (ICLR 2020 · 33 citations)
What has been tried
131 papers here report 1,197 measured changes (a component added, removed or swapped, compared with a baseline): helped 62%, no clear effect 10%, hurt 23%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| skip middle layers | Llama2 benchmarks · normalized median | Hurt | Transformer Layers as Painters (AAAI 2025 · 10 citations) | |
| Train count examples across a range of lengths instead of using sequences all of the same length | Length generalization · exact match | Helped | What Algorithms can Transformers Learn? A Study in Length Generalization (ICLR 2024 · 8 citations) | |
| Apply laziness to MHSA only rather than to MHSA and Feedforward together | ImageNet 256×256, DDIM 20 steps · generation performance | Hurt | LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers (AAAI 2025 · 6 citations) | |
| Replace Softmax attention with scaled-σ attention | WikiText-103 · perplexity | 18.98 -> 19.07 | Hurt | Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption (ICML 2024 · 6 citations) |
| Use an 8-layer recurrent block with 2 recurrences instead of a non-recurrent 16-layer model | Addition, OOD · error rate | half the error | Helped | Transformers Can Do Arithmetic with the Right Embeddings (NeurIPS 2024 · 6 citations) |
| Add the negative-mover intervention alongside the inhibition-head intervention | Colored Objects · accuracy | 81.5% -> 93.7% | Helped | Circuit Component Reuse Across Tasks in Transformer Language Models (ICLR 2024 · 3 citations) |
| Use 20 warmup epochs | Main dataset · accuracy | 89.48 -> 87.65 | Hurt | Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems (ICML 2024 · 3 citations) |
| Use Hard-ALiBi positional encoding | Length generalization from strings ≤ 50 to strings up to 1000 · string-level accuracy | Helped | Repeat After Me: Transformers are Better than State Space Models at Copying (ICML 2024 · 3 citations) | |
| Use mixed representation instead of contextual representation | BiRD · Pearson correlation | 20.19 -> 27.72 | Helped | Towards Equipping Transformer with the Ability of Systematic Compositionality (AAAI 2024 · 2 citations) |
| Increase the number of attention heads | αβα vs. αββ template task · test loss | Helped | When can transformers reason with abstract symbols? (ICLR 2024 · 2 citations) | |
| add the log transform to relative distances | C4 validation, sequence length 8192 · log perplexity | 3.334 -> 3.085 | Helped | Functional Interpolation for Relative Positions improves Long Context Transformers (ICLR 2024 · 2 citations) |
| adapt attention and positional bias jointly instead of using f(B) alone | Books, train 512, eval 8192 · perplexity | 23.49 -> 17.88 | Helped | DAPE: Data-Adaptive Positional Encoding for Length Extrapolation (NeurIPS 2024 · 2 citations) |
What reviewers will expect
Datasets and baselines named in the 90 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet-1k · 5 of 90 papers
- ImageNet · 3 of 90 papers
- OpenWebText · 3 of 90 papers
- Countdown · 2 of 90 papers
- CIFAR-100 · 2 of 90 papers
- CIFAR-10 · 2 of 90 papers
- MNIST · 2 of 90 papers
Baselines
- MLP · 4 of 90 papers
- NoPE · 3 of 90 papers
- LSTM · 3 of 90 papers
- RoPE · 3 of 90 papers
- PonderNet · 2 of 90 papers
- FIRE · 2 of 90 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 199 papers; from OpenAlex and the papers' affiliations).
- New York University · 12 papers
- Meta · 10 papers
- Carnegie Mellon University · 9 papers
- Microsoft · 9 papers
- Massachusetts Institute of Technology · 9 papers
- The University of Hong Kong · 7 papers
- Google · 7 papers
- University of Pennsylvania · 6 papers
- Stanford University · 6 papers
- IBM · 4 papers
Use it
Download BibTeX of 38 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Attention head mechanisms (75 papers)
- Training dynamics of in-context learning (62 papers)
- Vision transformer design (30 papers)
Elsewhere in the topic
- State space and recurrent sequence models (172 papers)
- Sparse and structured attention (163 papers)
- Sparse attention for long-context LLMs (147 papers)
- Attention head and block redesign (127 papers)
- Efficient and robust vision transformers (124 papers)
- Efficient language transformers (105 papers)
- Language model pretraining and induction (99 papers)
- Hybrid and interpretable ViT designs (94 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).