Research map / Label-efficient and robust learning / Self-supervised and contrastive
Self-Guided Masked Autoencoder
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that MAE learns pattern-based patch clustering early in pre-training and uses its own encoder-derived patch clusters (via normalized cut and an exploitation-rate trigger) to generate object-centric informed masks, speeding up MAE training without external models.
Problem
Vanilla MAE uses random masking and it is not well understood what and how it learns; prior informed-masking methods rely on external pretrained models or labels.
Approach
MAE intrinsically learns patch-level clustering from very early in pre-training; the method uses its own patch similarity graph (normalized cut bi-partition plus relevance score) to mask the object-centric cluster, with the start time chosen by a decoder mask-token exploitation rate.
Key results
- Self-guided MAE reaches 62.9% linear probing on ImageNet-1K after 400 epochs vs 61.4% for MAE and 61.7% for AMT.
- Fine-tuning on ImageNet-1K gives 83.2 vs 82.5 (MAE); COCO AP box 43.3 vs 43.0 and ADE20K mIoU 45.2 vs 45.0.
- With 1600 epochs of pre-training, linear probing reaches 68.7 vs 68.0 for MAE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use self-guided informed masking instead of random masking | MAE · ImageNet-1K, linear probing, 400 pre-training epochs | accuracy: 61.4 -> 62.9 | Helped |
| use self-guided informed masking instead of random masking | MAE · ImageNet-1K, fine-tuning | accuracy: 82.5 -> 83.2 | Helped |
| use self-guided informed masking instead of random masking | MAE · iNaturalist 2019 | fine-tuning accuracy: 78.7 -> 78.9 | Helped |
| use self-guided informed masking instead of random masking | MAE · CIFAR-100 | fine-tuning accuracy: 89.3 -> 90.0 | Helped |
| use self-guided informed masking instead of random masking | MAE · CUB200-2011 | fine-tuning accuracy: 81.8 -> 82.8 | Helped |
| use self-guided informed masking instead of random masking | MAE · COCO | AP box: 43.0 -> 43.3 | Helped |
| use self-guided informed masking instead of random masking | MAE · COCO | AP mask: 38.9 -> 39.3 | Helped |
| use self-guided informed masking instead of random masking | MAE · ADE20K | mIoU: 45.0 -> 45.2 | Helped |
| use encoder layer 3 instead of encoder layer 11 to generate masks (ablation) | Self-Guided MAE · ImageNet-1K, linear probing | accuracy: 62.9 -> 62.3 | Hurt |
| remove hint tokens (ablation) | Self-Guided MAE · ImageNet-1K, linear probing | accuracy: 62.9 -> 52.3 | Hurt |
| use random hint selection rather than S_i-based selection (ablation) | Self-Guided MAE · ImageNet-1K, linear probing | accuracy: 62.5 -> 62.9 | Helped |
| mask the object cluster rather than the background cluster (ablation) | Self-Guided MAE · ImageNet-1K, linear probing | accuracy: 61.1 -> 62.9 | Helped |
Surprising findings
- Patch clustering (bi-partition gap) is already clear from very early pre-training epochs, and the decoder mask-token exploitation rate surpasses visible-token rate around epoch 50.
- Removing hint tokens drops linear probing from 62.9 to 52.3.
Limitations
- Less improvement when images are excessively fragmented into many clusters (e.g., some segmentation datasets), where informed masks resemble random masking.
- Relies on bi-partitioning from the model's own early-stage features, which may be imperfect in early epochs.
Experimental setup
Datasets: ImageNet-1K CIFAR-100 iNaturalist 2019 CUB-200-2011 COCO ADE20K
Benchmarks: ImageNet-1K iNaturalist 2019 CIFAR-100 CUB-200-2011 COCO ADE20K
Metrics: top-1 accuracy (linear probing and fine-tuning) AP box AP mask mIoU feature variance similarity variance NMI attention distance reconstruction MSE
Baselines: MAE AMT SemMAE HPM CL-MAE MoCo ViT
Builds on: MAE Attention-driven Masking and Throwing (AMT) Normalized Cut hierarchical latent variable model of MAE attention rollout
Closest papers by meaning
- Learning Mask Invariant Mutual Information for Masked Image Modeling (ICLR 2025 · similarity 0.92)
- The Dynamic Duo of Collaborative Masking and Target for Advanced Masked Autoencoder Learning (AAAI 2025 · similarity 0.90)
- Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget (AAAI 2024 · similarity 0.90)
- R-MAE: Regions Meet Masked Autoencoders (ICLR 2024 · similarity 0.90)
- Self-Guided Masked Autoencoders for Domain-Agnostic Self-Supervised Learning (ICLR 2024 · similarity 0.89)
- Exploring Target Representations for Masked Autoencoders (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.