Research map / Label-efficient and robust learning / Self-supervised and contrastive
Masked autoencoder pretraining
Improves masked autoencoders using mean teachers, curricula and frequency-aware or compressed designs.
48 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2022–2026) · an approach in Self-supervised and contrastive, keyword cluster “supervised ssl · representations · masked”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 1 of 2 tracked benchmarks.
25 papers in 2023–24 and 19 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 7.7% to 5.9%.
Open questions the data points to
- ADE20K (mIoU): best went from 48.6 (up to 2024) to 49.3 in 2025 — Learning Mask Invariant Mutual Information for Masked Image Modeling (ICLR 2025 · 0 citations)
- Papers report MAE on COCO (AP box) anywhere from 43 to 51.2: comparisons against it depend on whose number you take.
What looks settled
- ImageNet-1K (Top-1 accuracy): best up to 2024 87.4, best in 2025–26 85.7 from 4 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2022 · Masked Autoencoders that Listen (NeurIPS 2022 · 103 citations)
- 2023 · On Data Scaling in Masked Image Modeling (CVPR 2023 · 56 citations)
- 2024 · Exploring Target Representations for Masked Autoencoders (ICLR 2024 · 16 citations)
- 2025 · FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learning (CVPR 2025 · 11 citations)
- 2026 · Standing on the Shoulders of Giants: Rethinking EEG Foundation Model Pretraining via Multi-Teacher Distillation (NeurIPS 2026 · 0 citations)
Earliest papers here
- Masked Autoencoders that Listen (NeurIPS 2022 · 103 citations)
- Adversarial Masking for Self-Supervised Learning (ICML 2022 · 14 citations)
- Adapting Self-Supervised Vision Transformers by Probing Attention-Conditioned Masking Consistency (NeurIPS 2022 · 7 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
ImageNet-1K · Top-1 accuracy (higher is better)
6 papers, 2023–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 83.1 | Masked Frequency Modeling for Self-Supervised Visual Pre-Training (ICLR 2023 · 29 citations) |
| 2024 | 87.4 | Exploring Target Representations for Masked Autoencoders (ICLR 2024 · 16 citations) |
ADE20K · mIoU (higher is better)
4 papers, 2023–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 48.6 | Masked Frequency Modeling for Self-Supervised Visual Pre-Training (ICLR 2023 · 29 citations) |
| 2025 | 49.3 | Learning Mask Invariant Mutual Information for Masked Image Modeling (ICLR 2025 · 0 citations) |
What has been tried
37 papers here report 464 measured changes (a component added, removed or swapped, compared with a baseline): helped 70%, no clear effect 7%, hurt 20%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use 3 stages instead of 2 for the same 1600 total epochs | ImageNet-1K validation · top-1 accuracy | 84.5 -> 84.4 | Hurt | Exploring Target Representations for Masked Autoencoders (ICLR 2024 · 16 citations) |
| Add asymmetric augmented SSL instead of conventional contrastive learning | CUB200-2011, 48-bit · mAP (%) | 15.36 -> 26.57 | Helped | An Asymmetric Augmented Self-Supervised Learning Method for Unsupervised Fine-Grained Image Hashing (CVPR 2024 · 15 citations) |
| Add NNCLR head initialization on frozen encoder features before contrastive tuning | ImageNet k-NN · accuracy | 68.1 -> 77.4 | Helped | Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget (AAAI 2024 · 13 citations) |
| Replace random masking with CRFR-P facial masking | DfD and FAS · Video-level AUC | 79.51 -> 81.58 | Helped | FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learning (CVPR 2025 · 11 citations) |
| use encoder layer 3 instead of encoder layer 11 to generate masks | ImageNet-1K, linear probing · accuracy | 62.9 -> 62.3 | Hurt | Self-Guided Masked Autoencoder (NeurIPS 2024 · 5 citations) |
| Increase mask ratio from 0.4 to 0.75 | ImageNet-1K · Top-1 accuracy (%) | 83.1 -> 83.4; 83.6 -> 83.9 | Helped | Wavelet-Driven Masked Image Modeling: A Path to Efficient Visual Representation (AAAI 2025 · 2 citations) |
| Use a fixed temperature instead of annealing | ImageNet-1K · nearest-neighbor accuracy | 41.57 -> 47.40 | Hurt | From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modeling (CVPR 2025 · 2 citations) |
| Add clustering to MAE backbone features | MAFL same-identity matching · Mean Pixel Error | 0.55 -> 0.30 | Helped | SCE-MAE: Selective Correspondence Enhancement with Masked Autoencoder for Self-Supervised Landmark Estimation (CVPR 2024 · 2 citations) |
| Replace masked sub-branch with dropout (0.2) | ImageNet-1k · Top-1 accuracy | +0.2; +0.3 | Helped | Masking meets Supervision: A Strong Learning Alliance (CVPR 2025 · 1 citation) |
| Use K=10 rather than the reported best moderate K=50 for ClusterMIM Pixel | ImageNet-100 · LP Acc. | 52.7 -> 42.7 | Hurt | On the Role of Discrete Tokenization in Visual Representation Learning (ICLR 2024 · 1 citation) |
| add collaborative masking to the baseline | ImageNet-1K linear probing · top-1 accuracy | 68.0 to 73.5 | Helped | The Dynamic Duo of Collaborative Masking and Target for Advanced Masked Autoencoder Learning (AAAI 2025 · 0 citations) |
| Add max-MI loss | ImageNet-1K · accuracy | 82.2 -> 82.5 | Helped | Learning Mask Invariant Mutual Information for Masked Image Modeling (ICLR 2025 · 0 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| MAE | ADE20K | mIoU | 45 – 48.1 | 3 papers, e.g. NeurIPS 2024 and ICLR 2023 |
| MAE | COCO | AP box | 43 – 51.2 | 2 papers, e.g. NeurIPS 2024 and ICLR 2025 |
| MAE | COCO | AP mask | 38.9 – 45.5 | 2 papers, e.g. NeurIPS 2024 and ICLR 2025 |
What reviewers will expect
Datasets and baselines named in the 10 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet-1K · 7 of 10 papers
- ADE20K · 5 of 10 papers
- COCO · 2 of 10 papers
- Flowers102 · 2 of 10 papers
Baselines
- MAE · 7 of 10 papers
- DINO · 5 of 10 papers
- MoCo v3 · 5 of 10 papers
- BEiT · 4 of 10 papers
- iBOT · 3 of 10 papers
- SimMIM · 3 of 10 papers
- AttMask · 2 of 10 papers
- SemMAE · 2 of 10 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 44 papers; from OpenAlex and the papers' affiliations).
- Huawei · 3 papers
- Peking University · 3 papers
- Chinese Academy of Sciences · 2 papers
- University of Chinese Academy of Sciences · 2 papers
- Shanghai Jiao Tong University · 2 papers
- Microsoft Research Asia · 2 papers
- ByteDance · 2 papers
- Megvii · 2 papers
- University of Toronto · 2 papers
- Microsoft · 2 papers
Use it
Download BibTeX of 36 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Invariance-based self-supervised learning (168 papers)
- Data-efficient and robust self-supervision (122 papers)
- Self-supervised speech representations (24 papers)
Elsewhere in the topic
- Pretext-task visual pre-training (136 papers)
- Self-supervision in supervised pipelines (121 papers)
- Self-supervised neuroimaging representations (98 papers)
- Contrastive versus non-contrastive theory (74 papers)
- Inductive bias and minimal representations (66 papers)
- Self-supervised speech and brain decoding (56 papers)
- Empirical contrastive visual learning (38 papers)
- Representation evaluation and trade-offs (37 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).