Research map / Architectures and efficiency / Attention and transformers
Transformer expressivity and circuits
Analyzes Turing completeness, induction-head representability, and attention-head stability in transformers.
56 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Attention and transformers, keyword cluster “transformers · language · icl”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
16 papers in 2023–24 and 19 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 2.7% to 1.6%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Structured Attention Networks (ICLR 2017 · 98 citations)
- 2019 · Universal Transformers (ICLR 2019 · 382 citations)
- 2020 · Neural Execution Engines: Learning to Execute Subroutines (NeurIPS 2020 · 13 citations)
- 2021 · R-Drop: Regularized Dropout for Neural Networks (NeurIPS 2021 · 305 citations)
- 2022 · Bridging Global Context Interactions for High-Fidelity Image Completion (CVPR 2022 · 118 citations)
- 2023 · Masked Image Modeling With Local Multi-Scale Reconstruction (CVPR 2023 · 58 citations)
- 2024 · Sharingan: A Transformer Architecture for Multi-Person Gaze Following (CVPR 2024 · 31 citations)
- 2025 · Robust Tracking via Mamba-based Context-aware Token Learning (AAAI 2025 · 27 citations)
- 2026 · Thinking in Scales: Accelerating Gigapixel Pathology Image Analysis via Adaptive Continuous Reasoning (ICML 2026 · 0 citations)
Earliest papers here
- Structured Attention Networks (ICLR 2017 · 98 citations)
- Attention is All you Need (NeurIPS 2017)
- Universal Transformers (ICLR 2019 · 382 citations)
What has been tried
48 papers here report 517 measured changes (a component added, removed or swapped, compared with a baseline): helped 67%, no clear effect 9%, hurt 20%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| use the location-aware gaze token instead of head-mask embeddings | GazeFollow · Avg. D. | 0.117 -> 0.113 | Helped | Sharingan: A Transformer Architecture for Multi-Person Gaze Following (CVPR 2024 · 31 citations) |
| add track token | LaSOT · AUC | 71.1 -> 71.4 | Helped | Robust Tracking via Mamba-based Context-aware Token Learning (AAAI 2025 · 27 citations) |
| pre-train rather than train from scratch | CrossPlatform (Android) · accuracy | 88.08% -> 93.00% | Helped | MIETT: Multi-Instance Encrypted Traffic Transformer for Encrypted Traffic Classification (AAAI 2025 · 20 citations) |
| Use cross-scale tokenization with K=3 instead of single-scale tokenization with K=1 | Four representative tasks: emotion recognition, motor imagery, event classification, imagined speech · performance | 10.8%, 20.7%, and 12.1% | Helped | CSBrain: A Cross-scale Spatiotemporal Brain Foundation Model for EEG Decoding (NeurIPS 2025 · 6 citations) |
| Add N-gram Transformer encoder (NG-Midiformer) on UCW-7 tokens vs MIDI-Bert UCW-7 | Pianist8 / EMOPIA / GTZAN / Nottingham · accuracy (%) | +14.33, +9.08, +16.31, +15.79 (UCW-7, composer/emotion/genre/dance) | Helped | N-gram Unsupervised Compoundation and Feature Injection for Better Symbolic Music Understanding (AAAI 2024 · 6 citations) |
| Use Du-IN MAE initialization instead of training from scratch | Chinese word-reading sEEG, 61-word classification · accuracy | 56.29±5.20 -> 62.70±4.69 | Helped | Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals (NeurIPS 2024 · 6 citations) |
| add CCA loss to the evenly grouped channel-wise model | Kodak · BD-rate | -13.31% -> -14.72% | Helped | Causal Context Adjustment Loss for Learned Image Compression (NeurIPS 2024 · 4 citations) |
| use criss-cross attention instead of full attention | emotion recognition and motor imagery datasets · performance | Helped | CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding (ICLR 2025 · 3 citations) | |
| Add key-value cross-attention | EP test set · Micro F1 (%) | 80.74 -> 84.38 | Helped | Harnessing Event Sensory Data for Error Pattern Prediction in Vehicles: A Language Model Approach (AAAI 2025 · 2 citations) |
| Use attribution fit with α_train=0.05 rather than α_train=0.10 | ImageNet ResNet-50, small test ablation fractions · Pearson correlation | Helped | Decomposing and Editing Predictions by Modeling Model Computation (ICML 2024 · 2 citations) | |
| add CSCA | IXI (8×) radial · PSNR | 40.767 -> 41.948 | Helped | Breaking Information Isolation: Accelerating MRI via Inter-sequence Mapping and Progressive Masking (AAAI 2025 · 1 citation) |
| add dynamic semantic-aware modeling instead of standard Transformer correlation modeling | DTB70 · precision | 87.7 -> 91.2 | Helped | Dynamic Semantic-Aware Correlation Modeling for UAV Tracking (NeurIPS 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 13 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- COCO · 2 of 13 papers
- LaSOT · 2 of 13 papers
- TrackingNet · 2 of 13 papers
- GOT-10k · 2 of 13 papers
- TUEG · 2 of 13 papers
- PhysioNet-MI · 2 of 13 papers
- SHU-MI · 2 of 13 papers
- FACED · 2 of 13 papers
Baselines
- AQATrack · 2 of 13 papers
- OSTrack · 2 of 13 papers
- EEGNet · 2 of 13 papers
- EEGConformer · 2 of 13 papers
- SPaRCNet · 2 of 13 papers
- ContraWR · 2 of 13 papers
- CNN-Transformer · 2 of 13 papers
- FFCL · 2 of 13 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 35 papers; from OpenAlex and the papers' affiliations).
- Wuhan University · 2 papers
- Nanjing University · 2 papers
- Google · 2 papers
- Google DeepMind · 2 papers
- Massachusetts Institute of Technology · 2 papers
Use it
Download BibTeX of 37 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Transformer block and depth design (89 papers)
- Efficient language transformers (105 papers)
- Language model pretraining and induction (99 papers)
Elsewhere in the topic
- Transformer expressivity and learnability (217 papers)
- State space and recurrent sequence models (172 papers)
- Sparse and structured attention (163 papers)
- Sparse attention for long-context LLMs (147 papers)
- Attention head and block redesign (127 papers)
- Efficient and robust vision transformers (124 papers)
- Hybrid and interpretable ViT designs (94 papers)
- KV cache eviction and compression (92 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).