Research map / Label-efficient and robust learning / Self-supervised and contrastive
Exploring Target Representations for Masked Autoencoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that the choice of teacher target in masked knowledge distillation matters little after multi-stage distillation, and proposes dBOT, which bootstraps a randomly initialized teacher across stages to pre-train ViTs without extra pre-training.
Problem
Masked image modeling methods rely on carefully chosen target representations from pre-trained teachers (DALL-E, CLIP, DINO, etc.), which add pre-training cost and design effort; the paper asks whether the teacher design matters for masked knowledge distillation.
Approach
Across teachers (DeiT, DINO, DALL-E, MAE, random), students converge to similar performance and properties after multiple distillation stages; so dBOT starts from a random teacher and repeatedly assigns the student's weights to the teacher at breakpoints (momentum 0 at breakpoints, 1 otherwise), re-initializing the student each stage.
Key results
- dBOT reaches 84.5%, 86.6%, and 87.4% ImageNet-1K top-1 fine-tuning accuracy with ViT-B, ViT-L, and ViT-H (88.0% at 448 resolution), above MAE's 83.6%, 85.9%, 86.9%, 87.8%.
- dBOT achieves 52.7 AP box with ViT-B and 56.0 with ViT-L on COCO with Cascade Mask R-CNN, and 49.5 and 54.5 mIoU on ADE20K with ViT-B and ViT-L.
- Performance variance across teachers in classification drops from 2.24 after stage 0 to 0.04 after stage 3, and a random teacher reaches 84.5% after 2 stages.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dBOT instead of MAE pretraining | ViT-B · ImageNet-1K | top-1 accuracy: 83.6 -> 84.5 | Helped |
| Use dBOT instead of MAE pretraining | ViT-L · ImageNet-1K | top-1 accuracy: 85.9 -> 86.6 | Helped |
| Use dBOT instead of MAE pretraining | ViT-B and ViT-L · COCO, Cascade Mask R-CNN | AP box: +2.1 points; +2.0 points | Helped |
| Use dBOT instead of MAE pretraining | ViT-B and ViT-L · ADE20K, UperNet | mIoU: 48.1 -> 49.5; 53.6 -> 54.5 | Helped |
| Replace a pre-trained teacher with a randomly initialized teacher and bootstrap across stages | dBOT · ImageNet-1K classification, COCO detection, ADE20K segmentation | performance: 84.5; 52.7; 49.5 | No clear effect |
| Use 3 stages instead of 2 for the same 1600 total epochs (ablation) | dBOT, ViT-B/16 · ImageNet-1K validation | top-1 accuracy: 84.5 -> 84.4 | Hurt |
| Use 1 stage of 1600 epochs instead of 2 stages (ablation) | dBOT, ViT-B/16 · ImageNet-1K validation | top-1 accuracy: 83.6 -> 84.5 | Hurt |
| Allocate 800 epochs to each of two stages (ablation) | dBOT, ViT-B/16 · ImageNet-1K validation | top-1 accuracy: 84.3 -> 84.5 | Helped |
| Use explicit stage-wise teacher updates instead of EMA momentum (ablation) | dBOT, ViT-B/16 · ImageNet-1K validation | top-1 accuracy: 83.6 -> 84.5; 83.9 -> 84.5; 82.1 -> 84.5 | Helped |
| Use target patch representations without LayerNorm (ablation) | dBOT, ViT-B/16 · ImageNet-1K validation | top-1 accuracy: 84.3 -> 84.5 | Helped |
| Reinitialize the student at stage breakpoints (ablation) | dBOT, ViT-B/16 · ImageNet-1K validation | top-1 accuracy: 84.2 -> 84.5 | Helped |
| Use a mask ratio of 0.75 (ablation) | dBOT, ViT-B/16 · ImageNet-1K validation | top-1 accuracy: 84.3 -> 84.5; 84.2 -> 84.5 | Helped |
Surprising findings
- A randomly initialized teacher, after multi-stage bootstrapped distillation, matches the performance of pre-trained teachers such as DINO and MAE.
- Using patch tokens from a random network as the reconstruction target performs comparably to raw pixel targets in MAE (83.6% vs 83.6% at 1600 epochs).
Limitations
- Results with CLIP as teacher exceed those from random teachers, contradicting the main conclusion when extra data is used; the paper attributes this to CLIP's 400M image-text pairs.
- The bootstrapped pipeline needs multiple pre-training stages, so total compute is larger than a single stage, and the default setting uses 2 or 3 stages of 800 epochs each.
Experimental setup
Datasets: ImageNet-1K COCO ADE20K CIFAR10 CIFAR100 iNaturalist18 iNaturalist19 Flowers Cars Pascal VOC 2012
Benchmarks: ImageNet-1K COCO ADE20K CIFAR10 CIFAR100 iNaturalist18 iNaturalist19 Flowers Cars Pascal VOC 2012
Metrics: top-1 accuracy AP box AP mask mIoU mAcc CorLoc linear probing accuracy semi-supervised accuracy
Baselines: MAE BEiT iBOT data2vec DINO MoCo v3 supervised DeiT MaskFeat
Builds on: MAE BEiT iBOT data2vec Cascade Mask R-CNN
Closest papers by meaning
- Asymmetric Masked Distillation for Pre-Training Small Foundation Models (CVPR 2024 · similarity 0.90)
- The Dynamic Duo of Collaborative Masking and Target for Advanced Masked Autoencoder Learning (AAAI 2025 · similarity 0.89)
- Self-Guided Masked Autoencoder (NeurIPS 2024 · similarity 0.89)
- Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners (ICLR 2024 · similarity 0.89)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.89)
- Self-Guided Masked Autoencoders for Domain-Agnostic Self-Supervised Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.