Research map / Architectures and efficiency / Attention and transformers
Transformer block and depth design
Reworks transformer layers with depth-weighted connections, adaptive depth, and dynamical analyses.
89 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2020–2026) · an approach in Attention and transformers, keyword cluster “transformers · language · icl”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
33 papers in 2023–24 and 34 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 5.7% to 2.9%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2020 · Reducing Transformer Depth on Demand with Structured Dropout (ICLR 2020 · 371 citations)
- 2021 · Hash Layers For Large Sparse Models (NeurIPS 2021 · 48 citations)
- 2022 · Reduce Information Loss in Transformers for Pluralistic Image Inpainting (CVPR 2022 · 104 citations)
- 2023 · Generalized Relation Modeling for Transformer Tracking (CVPR 2023 · 247 citations)
- 2024 · Simplifying Transformer Blocks (ICLR 2024 · 10 citations)
- 2025 · Kolmogorov-Arnold Transformer (ICLR 2025 · 16 citations)
- 2026 · Infinite-Precision Autoregressive Modeling for Vector Graphics and Layouts (ICML 2026 · 0 citations)
Earliest papers here
- Reducing Transformer Depth on Demand with Structured Dropout (ICLR 2020 · 371 citations)
- Stabilizing Transformers for Reinforcement Learning (ICML 2020 · 130 citations)
- Sketchformer: Transformer-Based Representation for Sketched Structure (CVPR 2020 · 114 citations)
What has been tried
71 papers here report 769 measured changes (a component added, removed or swapped, compared with a baseline): helped 67%, no clear effect 8%, hurt 23%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use KAT-T with learnable group-rational activations instead of GELU | ImageNet-1K val · Top-1 accuracy | 72.7 -> 74.6; 1.9% | Helped | Kolmogorov-Arnold Transformer (ICLR 2025 · 16 citations) |
| remove normalisation layers from SAS-P | GLUE · GLUE | Hurt | Simplifying Transformer Blocks (ICLR 2024 · 10 citations) | |
| Shift cache steps from [2,4,6,8,10,...] to [3,5,7,9,11,...] | ImageNet 256×256, 20 NFE · FID | 5.30 -> 2.80 | Helped | Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching (NeurIPS 2024 · 10 citations) |
| Increase input standard deviation from 0.3 to 2.5 | ImageNet 128, 100,000 iterations · FID-10k | 65.38 -> 47.04 | Helped | FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification (NeurIPS 2024 · 10 citations) |
| combine STB and CTB in dual branches rather than use all STB | Urban100, Gaussian color σ=50 · PSNR (dB) | 29.87 -> 29.94 | Helped | Xformer: Hybrid X-Shaped Transformer for Image Denoising (ICLR 2024 · 7 citations) |
| increase the difference threshold above 0.1 | action recognition · accuracy | Hurt | Don't Look Twice: Faster Video Transformers with Run-Length Tokenization (NeurIPS 2024 · 7 citations) | |
| Add MoD token routing with a uniform compression ratio | LAION-5B, 10K images · FID | 11.93 -> 22.78 | Hurt | Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers (CVPR 2025 · 4 citations) |
| add lightweight fine-tuning to Nimbus | GLUE Avg. · average score | 82.53 -> 83.04 | Helped | Nimbus: Secure and Efficient Two-Party Inference for Transformers (NeurIPS 2024 · 3 citations) |
| Add the entropy loss to budget loss | ImageNet-1k validation · within-image computation diversity | Helped | Adaptive Computation Modules: Granular Conditional Computation for Efficient Inference (AAAI 2025 · 2 citations) | |
| Add AdaLN channel modulation to DiT features | Semantic correspondence across datasets · PCK | more than 20% | Helped | Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations (NeurIPS 2025 · 2 citations) |
| use MCP rather than the vanilla attention penalty | AGNEWS, TextFooler · AUA% | Helped | ProTransformer: Robustify Transformers via Plug-and-Play Paradigm (NeurIPS 2024 · 2 citations) | |
| use softmax normalization rather than linear-norm normalization for MLP attribution | Known_1000 · comprehensiveness (patch-top, 0.1) | 64.8 -> 80.8 | Helped | DePass: Unified Feature Attributing by Simple Decomposed Forward Pass (NeurIPS 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 28 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet-1K · 5 of 28 papers
- C4 · 4 of 28 papers
- Wikitext-103 · 3 of 28 papers
- CIFAR-10 · 2 of 28 papers
- OpenWebText · 2 of 28 papers
- ImageNet · 2 of 28 papers
- Wikipedia · 2 of 28 papers
- ADE20K · 2 of 28 papers
Baselines
- Transformer · 3 of 28 papers
- Hyper-Connections · 2 of 28 papers
- ViT · 2 of 28 papers
- Mix-LN · 2 of 28 papers
- DeiT · 2 of 28 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 67 papers; from OpenAlex and the papers' affiliations).
- Shanghai Jiao Tong University · 4 papers
- Microsoft · 4 papers
- Adobe · 3 papers
- Carnegie Mellon University · 3 papers
- University of California, Berkeley · 3 papers
- Google · 3 papers
- Google DeepMind · 3 papers
- Princeton University · 3 papers
- National University of Singapore · 3 papers
- Alibaba Group · 3 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Efficient language transformers (105 papers)
- Language model pretraining and induction (99 papers)
- Transformer expressivity and circuits (56 papers)
Elsewhere in the topic
- Transformer expressivity and learnability (217 papers)
- State space and recurrent sequence models (172 papers)
- Sparse and structured attention (163 papers)
- Sparse attention for long-context LLMs (147 papers)
- Attention head and block redesign (127 papers)
- Efficient and robust vision transformers (124 papers)
- Hybrid and interpretable ViT designs (94 papers)
- KV cache eviction and compression (92 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).