Research map / Label-efficient and robust learning / Self-supervised and contrastive
A Theoretical Framework for Masked Pretraining (MPT)
Qi Zhang, Runyu Zhou, Yifei Wang, Yisen Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add uniformity regularizer | MAE · ImageNet-100 (ViT-Base) linear probing | accuracy: 61.2 -> 67.5 | Improved |
| add uniformity regularizer | MAE · CIFAR-10 (ViT-Tiny) linear probing | accuracy: 59.6 -> 68.9 | Improved |
| add uniformity regularizer | MAE · ImageNet-1K (ViT-Base) linear probing | accuracy: 55.4 -> 58.5 | Improved |
| add uniformity regularizer | MAE · in-domain fine-tuning (CIFAR-10, ImageNet-100, ImageNet-1K) | accuracy: 89.6 -> 89.4 | No clear effect |
| add uniformity regularizer | MAE · iNaturalist-2021 fine-tuning | accuracy: 66.0 -> 69.4 | Improved |
| add uniformity regularizer | MAE · CUB and Cars fine-tuning | accuracy: 81.5 -> 81.8; 59.8 -> 60.3 | Improved |
| add uniformity regularizer | MAE · ImageNet-C Defocus Blur linear probing | accuracy: 10.48 -> 26.20 | Improved |
| add uniformity regularizer | MAE · ImageNet-A linear probing (frozen) | accuracy: 0.57 -> 0.68 | Improved |
| add uniformity loss | SimMIM · ImageNet-100 linear probing | accuracy: 54.3 -> 61.1 | Improved |
| set uniformity coefficient to 0.01 instead of 0 (ablation) | MAE (U-MPT) · ImageNet-100 linear probing | accuracy: 61.2 -> 67.5 | Improved |
| set uniformity coefficient to 1e-3 instead of 0 (ablation) | MAE (U-MPT) · ImageNet-100 linear probing | accuracy: 61.2 -> 65.9 | Improved |
| raise uniformity coefficient to 0.1 from 0.01 (ablation) | MAE (U-MPT) · ImageNet-100 linear probing | accuracy: 67.5 -> 45.1 | Hurt |
Closest papers by meaning
- Towards Understanding Why Mask Reconstruction Pretraining Helps in Downstream Tasks (ICLR 2023 · similarity 0.90)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.90)
- Video Language Model Pretraining with Spatio-temporal Masking (CVPR 2025 · similarity 0.89)
- On Masked Pre-training and the Marginal Likelihood (NeurIPS 2023 · similarity 0.89)
- Masking meets Supervision: A Strong Learning Alliance (CVPR 2025 · similarity 0.88)
- Exploring Target Representations for Masked Autoencoders (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
