Research map / Label-efficient and robust learning / Self-supervised and contrastive
From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a prototype-driven curriculum for MAE that clusters features, samples near-centroid images first, and anneals sampling temperature to expand the effective dataset, improving ImageNet-1K efficiency.
Problem
Masked image modeling suffers from optimization difficulties early in training, where models must map partial observations to the full complex image distribution before learning basic visual processing; this makes training slow and inefficient.
Approach
Use mini-batch K-means clusters in a self-supervised feature space (DINO) to identify prototypical images, then sample from the dataset with a temperature-controlled softmax over per-cluster normalized distances to centroids, annealing the temperature (via a scheduled effective data size) so training moves from prototypes to the full distribution.
Key results
- Prototypical curriculum reaches 47.40% NN accuracy at 800 epochs versus 30.25% for the MAE baseline at 800 epochs.
- At 200 epochs the curriculum reaches 34.92% NN and 63.74% LP, already above the 800-epoch MAE baseline (30.25% NN, 64.25% LP).
- Curriculum-trained MAE matches the baseline's 100-epoch-level performance at 100 epochs, reported as an 8x speedup; 5-shot probing on eight fine-grained datasets is comparable or better at 200/400 epochs than the 1600-epoch baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a temperature-annealed prototypical curriculum instead of uniform MAE sampling | MAE · ImageNet-1K | nearest-neighbor accuracy: 30.25 -> 47.40 | Helped |
| Use a temperature-annealed prototypical curriculum instead of uniform MAE sampling | MAE · ImageNet-1K | linear-probe accuracy: 64.25 -> 68.84 | Helped |
| Use a temperature-annealed prototypical curriculum instead of uniform MAE sampling | MAE · ImageNet-1K | fine-tuning accuracy: 83.08 -> 83.31 | Helped |
| Prioritize hard samples distant from cluster centroids | MAE · ImageNet-1K | nearest-neighbor accuracy: 28.73 -> 24.63 | Hurt |
| Use a fixed temperature instead of annealing (ablation) | Prototypical curriculum · ImageNet-1K | nearest-neighbor accuracy: 41.57 -> 47.40 | Hurt |
| Use a fixed temperature instead of annealing (ablation) | Prototypical curriculum · ImageNet-1K | linear-probe accuracy: 67.71 -> 68.84 | Hurt |
| Use a fixed temperature of 0.01 (ablation) | Prototypical curriculum · ImageNet-1K | nearest-neighbor accuracy: 5.69 | Hurt |
| Identify prototypes using DINO features instead of untrained ViT-B/16 features (ablation) | Prototypical curriculum · ImageNet-1K | nearest-neighbor accuracy: 32.63 -> 40.15 | Helped |
| Identify prototypes using DINO features instead of MAE-pretrained features (ablation) | Prototypical curriculum · ImageNet-1K | nearest-neighbor accuracy: 34.14 -> 40.15 | Helped |
| Identify prototypes using DINO features instead of supervised features (ablation) | Prototypical curriculum · ImageNet-1K | nearest-neighbor accuracy: 39.70 -> 40.15 | No clear effect |
| Use SIFT features for prototype identification (ablation) | Prototypical curriculum · ImageNet-1K | linear-probe accuracy: 69.17 | Helped |
| Use ViT-S/8 instead of ViT-B/16 as the DINO prototype-identification backbone (ablation) | Prototypical curriculum · ImageNet-1K | linear-probe accuracy: 68.73 -> 69.23 | Helped |
Surprising findings
- Hard-sample curriculum (prioritizing non-prototypical images) underperforms the MAE baseline on NN accuracy (24.63% vs 28.73%), contrary to supervised dataset-distillation findings.
- Clustering with ground-truth labels (oracle centroids) gives lower NN accuracy (37.83%) than self-supervised DINO clustering (41.05%).
Limitations
- Evaluation limited to ImageNet-1K with ViT-B/16 pre-training; scaling to larger datasets is untested.
- Ablations on the number of clusters and prototype feature spaces show sensitivity to design choices.
- Oracle-comparison and hard-sample findings are not fully explained.
Experimental setup
Datasets: ImageNet-1K CUB-200 SUN397 OxfordPets Caltech101 UCF101 DTD Flowers102 Food101
Benchmarks: ImageNet-1K
Metrics: nearest neighbor accuracy linear probing accuracy fine-tuning accuracy 5-shot linear probing accuracy
Baselines: MAE uniform sampling Hard-sample curriculum
Builds on: Masked Autoencoders (MAE) DINO Davies-Bouldin index Dataset selection with hard samples (Xia et al. dataset distillation)
Closest papers by meaning
- Wavelet-Driven Masked Image Modeling: A Path to Efficient Visual Representation (AAAI 2025 · similarity 0.89)
- Morphing Tokens Draw Strong Masked Image Models (ICLR 2025 · similarity 0.89)
- Self-Organizing Visual Prototypes for Non-Parametric Representation Learning (ICML 2025 · similarity 0.88)
- Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners (ICLR 2024 · similarity 0.88)
- Pre-training with Random Orthogonal Projection Image Modeling (ICLR 2024 · similarity 0.88)
- On the Role of Discrete Tokenization in Visual Representation Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.