Research map / Architectures and efficiency / Attention and transformers
Token merging and masked pretraining
Trains vision transformers with token labeling, masked image modeling, and distilled tokens.
82 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Attention and transformers, keyword cluster “self attention · attention mechanism · softmax”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Leveling off Several papers from 2025–26 still report on 1 tracked benchmark, but the best result has barely moved since 2024.
24 papers in 2023–24 and 42 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 4.1% to 3.6%.
What looks settled
- ImageNet-1K (top-1 accuracy): best up to 2024 87.1, best in 2025–26 85.5 from 3 papers, at most 0.2% better, if at all.
No longer reported by papers here: ImageNet-1K val (top-1 accuracy, last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Inverse Compositional Spatial Transformer Networks (CVPR 2017 · 178 citations)
- 2019 · Efficient Training of BERT by Progressively Stacking (ICML 2019 · 66 citations)
- 2021 · All Tokens Matter: Token Labeling for Training Better Vision Transformers (NeurIPS 2021 · 143 citations)
- 2022 · Masked Autoencoders Are Scalable Vision Learners (CVPR 2022 · 7,924 citations)
- 2023 · Revealing the Dark Secrets of Masked Image Modeling (CVPR 2023 · 116 citations)
- 2024 · ParameterNet: Parameters Are All You Need for Large-scale Visual Pretraining of Mobile Networks (CVPR 2024 · 101 citations)
- 2025 · Transformers without Normalization (CVPR 2025 · 157 citations)
- 2026 · Selective Coupling of Decoupled Informative Regions: Masked Attention Alignment for Data-Free Quantization of Vision Transformers (ICML 2026 · 0 citations)
Earliest papers here
- Inverse Compositional Spatial Transformer Networks (CVPR 2017 · 178 citations)
- Efficient Training of BERT by Progressively Stacking (ICML 2019 · 66 citations)
- All Tokens Matter: Token Labeling for Training Better Vision Transformers (NeurIPS 2021 · 143 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
ImageNet-1K · top-1 accuracy (higher is better)
10 papers, 2021–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 83.9 | ImageNet-21K Pretraining for the Masses (NeurIPS 2021 · 3 citations) |
| 2022 | 87.1 | SimMIM: A Simple Framework for Masked Image Modeling (CVPR 2022 · 1,236 citations) |
ADE20K · mIoU (higher is better)
6 papers, 2022–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 59.9 | SimMIM: A Simple Framework for Masked Image Modeling (CVPR 2022 · 1,236 citations) |
ImageNet-1K val · top-1 accuracy (higher is better)
6 papers, 2021–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 86.4 | All Tokens Matter: Token Labeling for Training Better Vision Transformers (NeurIPS 2021 · 143 citations) |
| 2022 | 87.8 | Masked Autoencoders Are Scalable Vision Learners (CVPR 2022 · 7,924 citations) |
CIFAR-100 · Top-1 accuracy (%) (higher is better)
4 papers, 2021–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 91.7 | ImageNet-21K Pretraining for the Masses (NeurIPS 2021 · 3 citations) |
What has been tried
54 papers here report 676 measured changes (a component added, removed or swapped, compared with a baseline): helped 68%, no clear effect 8%, hurt 21%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Replace tanh with identity, retaining ω | ImageNet-1K · accuracy | 82.5% -> 61.0% → failed | Hurt | Transformers without Normalization (CVPR 2025 · 157 citations) |
| Increase dynamic experts from 1 to 8 | ImageNet-1K val, ImageNet-1K pretraining · Top-1 accuracy | 76.3 -> 77.7 | Helped | ParameterNet: Parameters Are All You Need for Large-scale Visual Pretraining of Mobile Networks (CVPR 2024 · 101 citations) |
| Reduce dynamic filter basis dimension from N=4 to N=2 | ImageNet-1K · Top-1 accuracy | 83.2 -> 83.0 | Hurt | FFT-Based Dynamic Token Mixer for Vision (AAAI 2024 · 64 citations) |
| Add the CLS attention component A_class | FSC147 test · MAE | 13.67 -> 11.56; 15.44% | Helped | Vision Transformer Off-the-Shelf: A Surprising Baseline for Few-Shot Class-Agnostic Counting (AAAI 2024 · 29 citations) |
| increase the CLAP window from 2×2 to 4×4 | ADE20K · mIoU (%) | 46.4 -> 44.6 | Hurt | ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers (CVPR 2024 · 19 citations) |
| Remove the cross-layer sliding window and connect ganglion layers to all preceding layers | ImageNet-1K · accuracy | 83.5 -> 83.4 | Hurt | SparX: A Sparse Cross-Layer Connection Mechanism for Hierarchical Vision Mamba and Transformer Networks (AAAI 2025 · 13 citations) |
| Replace pre-appended prompts with SVP's input-dependent token-wise prompts | CIFAR, DTD, CUB, Flowers · accuracy | 5.2% | Helped | Selective Visual Prompting in Vision Mamba (AAAI 2025 · 9 citations) |
| use energy-based token protection instead of leaving tokens unprotected | Image-text retrieval, r=0.925 · Rsum | 568.9 -> 573.4 | Helped | Accelerating Transformers with Spectrum-Preserving Token Merging (NeurIPS 2024 · 8 citations) |
| add fragment modeling to the transformer baseline | MitoEM-R validation · AP | 73.0 -> 74.2 | Helped | Electron Microscopy Images as Set of Fragments for Mitochondrial Segmentation (AAAI 2024 · 7 citations) |
| Use pos_11 rather than pos_0 in the Last LN | ImageNet-1K · Top-1 accuracy (%) | 73.51 -> 73.30 | Hurt | Maximizing the Position Embedding for Vision Transformers with Global Average Pooling (AAAI 2025 · 4 citations) |
| replace decoupled embeddings with self-attention keys for grouping | ImageNet-1k classification · accuracy | Hurt | Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers (NeurIPS 2024 · 4 citations) | |
| Use inter-channel attention only instead of combining with intra-channel attention | biological downstream tasks · representation quality | Hurt | ChAda-ViT : Channel Adaptive Attention for Joint Representation Learning of Heterogeneous Microscopy Images (CVPR 2024 · 3 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| MoCo v3 | ImageNet-1K | fine-tuning top-1 accuracy | 79 – 83.2 | 2 papers, e.g. ICML 2023 and CVPR 2022 |
What reviewers will expect
Datasets and baselines named in the 26 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet-1K · 10 of 26 papers
- ADE20K · 3 of 26 papers
- CIFAR-10 · 3 of 26 papers
- CIFAR-100 · 2 of 26 papers
- COCO 2017 · 2 of 26 papers
- MS COCO · 2 of 26 papers
- DTD · 2 of 26 papers
- UCF101 · 2 of 26 papers
Baselines
- MAE · 4 of 26 papers
- iBOT · 3 of 26 papers
- ViT · 2 of 26 papers
- VMamba · 2 of 26 papers
- MSVMamba · 2 of 26 papers
- data2vec · 2 of 26 papers
- Vim · 2 of 26 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 66 papers; from OpenAlex and the papers' affiliations).
- University of Science and Technology of China · 4 papers
- Tsinghua University · 4 papers
- University of California, Berkeley · 3 papers
- Microsoft Research Asia · 2 papers
- Peking University · 2 papers
- The University of Hong Kong · 2 papers
- Apple · 2 papers
- Princeton University · 2 papers
- University of Virginia · 2 papers
Use it
Download BibTeX of 35 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Efficient and robust vision transformers (124 papers)
- Hybrid and interpretable ViT designs (94 papers)
- Theory of self-attention dynamics (43 papers)
- Vision transformer pruning (26 papers)
Elsewhere in the topic
- Transformer expressivity and learnability (217 papers)
- State space and recurrent sequence models (172 papers)
- Sparse and structured attention (163 papers)
- Sparse attention for long-context LLMs (147 papers)
- Attention head and block redesign (127 papers)
- Efficient language transformers (105 papers)
- Language model pretraining and induction (99 papers)
- KV cache eviction and compression (92 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).