Research map / Architectures and efficiency / Attention and transformers
Attention head and block redesign
Modifies transformer attention heads, mixing, and primal-dual views to improve learning and efficiency.
127 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2019–2026) · an approach in Attention and transformers, keyword cluster “kv · cache · long context”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 2 of 6 tracked benchmarks.
37 papers in 2023–24 and 51 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 6.3% to 4.4%.
Open questions the data points to
- ImageNet-1k (Top-1 accuracy): best went from 83 (up to 2024) to 85.8 in 2025 — Breaking the Low-Rank Dilemma of Linear Attention (CVPR 2025 · 12 citations)
- WikiText-103 (perplexity): best went from 15.92 (up to 2024) to 14.29 in 2025 — On the Role of Hidden States of Modern Hopfield Network in Transformer (NeurIPS 2025 · 0 citations)
No longer reported by papers here: IWSLT14 De-En (BLEU, last in 2024); ImageNet-1K val (Top-1 accuracy, last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2019 · A Tensorized Transformer for Language Modeling (NeurIPS 2019 · 98 citations)
- 2020 · Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring (ICLR 2020 · 167 citations)
- 2021 · Transformer in Transformer (NeurIPS 2021 · 1,022 citations)
- 2022 · Vision Transformer With Deformable Attention (CVPR 2022 · 950 citations)
- 2023 · Slide-Transformer: Hierarchical Vision Transformer With Local Self-Attention (CVPR 2023 · 105 citations)
- 2024 · Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers (CVPR 2024 · 22 citations)
- 2025 · Linear Attention Modeling for Learned Image Compression (CVPR 2025 · 19 citations)
- 2026 · Share Your Attention: Transformer Weight Sharing via Matrix-based Dictionary Learning (AAAI 2026 · 1 citation)
Earliest papers here
- A Tensorized Transformer for Language Modeling (NeurIPS 2019 · 98 citations)
- Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring (ICLR 2020 · 167 citations)
- On Identifiability in Transformers (ICLR 2020 · 76 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
ImageNet-1k · Top-1 accuracy (higher is better)
13 papers, 2022–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 81.11 | Improving Transformer with an Admixture of Attention Heads (NeurIPS 2022 · 1 citation) |
| 2023 | 82.2 | Stabilizing Transformer Training by Preventing Attention Entropy Collapse (ICML 2023 · 8 citations) |
| 2024 | 83 | Cached Transformers: Improving Transformers with Differentiable Memory Cachde (AAAI 2024 · 3 citations) |
| 2025 | 85.8 | Breaking the Low-Rank Dilemma of Linear Attention (CVPR 2025 · 12 citations) |
LRA average · accuracy (higher is better)
5 papers, 2021–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 59.39 | Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr\"om Method (NeurIPS 2021) |
| 2022 | 59.93 | Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost (NeurIPS 2022 · 2 citations) |
| 2023 | 60.4 | Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation (NeurIPS 2023 · 3 citations) |
| 2025 | 60.72 | PolaFormer: Polarity-aware Linear Attention for Vision Transformers (ICLR 2025 · 6 citations) |
WikiText-103 · perplexity (lower is better)
5 papers, 2022–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 30.8 | Flowformer: Linearizing Transformers with Conservation Flows (ICML 2022 · 32 citations) |
| 2023 | 28.21 | Mimetic Initialization of Self-Attention Layers (ICML 2023 · 1 citation) |
| 2024 | 15.92 | Graph Convolutions Enrich the Self-Attention in Transformers! (NeurIPS 2024 · 3 citations) |
| 2025 | 14.29 | On the Role of Hidden States of Modern Hopfield Network in Transformer (NeurIPS 2025 · 0 citations) |
ADE20K · mIoU (higher is better)
4 papers, 2021–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 43.6 | Transformer in Transformer (NeurIPS 2021 · 1,022 citations) |
IWSLT14 De-En · BLEU (higher is better)
4 papers, 2021–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 34.7 | Implicit Kernel Attention (AAAI 2021 · 6 citations) |
| 2022 | 34.71 | Improving Transformer with an Admixture of Attention Heads (NeurIPS 2022 · 1 citation) |
| 2024 | 36 | Cached Transformers: Improving Transformers with Differentiable Memory Cachde (AAAI 2024 · 3 citations) |
ImageNet-1K val · Top-1 accuracy (higher is better)
4 papers, 2022–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 84 | Vision Transformer With Deformable Attention (CVPR 2022 · 950 citations) |
| 2024 | 86 | QT-ViT: Improving Linear Attention in ViT with Quadratic Taylor Expansion (NeurIPS 2024 · 1 citation) |
What has been tried
102 papers here report 1,093 measured changes (a component added, removed or swapped, compared with a baseline): helped 65%, no clear effect 8%, hurt 22%, mixed 5%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add peripheral tokens | COCO-Search18 target-absent test · cNSS | 3.003 -> 3.103 | Helped | Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers (CVPR 2024 · 22 citations) |
| Use NOSE instead of random layer selection | ImageNet-1k · Top-1 accuracy | 79.9 -> 81.8 | Helped | MLP Can Be A Good Transformer Learner (CVPR 2024 · 20 citations) |
| Use 2,4,6,2 Bi-RWKV layers instead of 2,2,2,2 | Kodak · BD-rate | 0.00% -> -2.50% | Helped | Linear Attention Modeling for Learned Image Compression (CVPR 2025 · 19 citations) |
| Replace decoder cross-attention with ALR feed-forward networks | IWSLT2017, four translation directions · BLEU | 0.257 -> 0.115; 0.324 -> 0.130; 0.276 -> 0.109; 0.292 -> 0.115 | Hurt | Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · 17 citations) |
| add KV buffer augmentation | ImageNet-1K · Top-1 accuracy | 82.1(-0.7) | Helped | Breaking the Low-Rank Dilemma of Linear Attention (CVPR 2025 · 12 citations) |
| Use ASC only | 512x512 · Attn FLOPs | 50% | Helped | DiTFastAttn: Attention Compression for Diffusion Transformer Models (NeurIPS 2024 · 11 citations) |
| Replace MLP with ISTA | CIFAR10, 400 epochs · accuracy | 91.2 -> 87.4 | Hurt | Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · 7 citations) |
| use linear positional encoding instead of no positional encoding | ImageNet-1K · top-1 accuracy | 83.78 -> 84.72 | Helped | SeTformer Is What You Need for Vision and Language (AAAI 2024 · 7 citations) |
| use trainable data-dependent attention patterns instead of fixed local patterns | Vchitect-T2V-Dataverse · loss | 0.13330 -> 0.13162 | Helped | Faster Video Diffusion with Trainable Sparse Attention (NeurIPS 2025 · 6 citations) |
| Vary the power-function scaling factor α across 3, 5, and 7 | LRA · Accuracy (%) | 60.72; 60.47; 60.57 | Mixed | PolaFormer: Polarity-aware Linear Attention for Vision Transformers (ICLR 2025 · 6 citations) |
| apply temperature scaling to queries only | Wikitext (fine-tune) · perplexity | 28.781 -> 27.416 | Helped | Selective Attention: Enhancing Transformer through Principled Context Control (NeurIPS 2024 · 6 citations) |
| replace isotropic Conv3x3 architecture with U-Net hourglass | ImageNet 256×256, 200K, CFG=1.5 · FID | 29.31 -> 14.65 | Helped | DiC: Rethinking Conv3x3 Designs in Diffusion Models (CVPR 2025 · 4 citations) |
What reviewers will expect
Datasets and baselines named in the 40 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet-1K · 13 of 40 papers
- CIFAR-100 · 7 of 40 papers
- CIFAR10 · 6 of 40 papers
- ADE20K · 5 of 40 papers
- Long-Range Arena · 4 of 40 papers
- COCO · 3 of 40 papers
- OpenWebText · 3 of 40 papers
- WikiText · 3 of 40 papers
Baselines
- ViT · 5 of 40 papers
- DeiT · 3 of 40 papers
- Performer · 3 of 40 papers
- Set Transformer · 2 of 40 papers
- Swin Transformer · 2 of 40 papers
- FLatten Transformer · 2 of 40 papers
- BiFormer · 2 of 40 papers
- Efficient Attention · 2 of 40 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 88 papers; from OpenAlex and the papers' affiliations).
- Google · 4 papers
- Shanghai Jiao Tong University · 4 papers
- Tsinghua University · 4 papers
- Massachusetts Institute of Technology · 4 papers
- Huawei · 4 papers
- ETHZ - ETH Zurich · 4 papers
- The University of Hong Kong · 3 papers
- University of Michigan - Ann Arbor · 3 papers
- IBM · 3 papers
- Inria · 2 papers
Use it
Download BibTeX of 37 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Sparse and structured attention (163 papers)
- Efficient image transformers (84 papers)
- Long-context KV and memory attention (88 papers)
- Transformer neural operators for PDEs (69 papers)
Elsewhere in the topic
- Transformer expressivity and learnability (217 papers)
- State space and recurrent sequence models (172 papers)
- Sparse attention for long-context LLMs (147 papers)
- Efficient and robust vision transformers (124 papers)
- Efficient language transformers (105 papers)
- Language model pretraining and induction (99 papers)
- Hybrid and interpretable ViT designs (94 papers)
- KV cache eviction and compression (92 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).