Research map / Label-efficient and robust learning / Recognition and long-tail
Patch-Aware Sample Selection for Efficient Masked Image Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Patch-Aware Sample Selection (PASS), combining a dynamically trained mask predictor and weighted patch-loss selection score, to cut MIM pre-training cost by selecting informative samples (MAE and simMIM on ImageNet-1K with 37% data budget).
Problem
Sample selection to reduce MIM pre-training cost performs poorly because random masking makes per-sample selection scores inconsistent and simple averaging of patch losses causes score homogeneity.
Approach
Use a low-cost mask predictor, trained for only a few epochs before each selection round, to consistently mask the highest-loss patches when scoring samples, and weight the masked patch losses by predicted informativeness rather than averaging them.
Key results
- PASS improves MAE finetune accuracy on ImageNet-1K by 0.7% (82.9 vs 82.2) using 37% data budget and ~0.59x pre-training time.
- PASS achieves ~1.7x speedup on MAE and ~2.2x on simMIM relative to the original methods, per the paper's claims.
- On ADE20K with UperNet, PASS improves MAE mIoU by +1.64 (42.28 vs 40.64) with both DTMP and WSS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PASS with a 37% data budget instead of full-data MAE pre-training | MAE · ImageNet-1K | finetune Top-1 accuracy (%): 82.2 -> 82.9 | Helped |
| Use PASS instead of basic sample selection | MAE · ImageNet-1K | linear probing Top-1 accuracy (%): 48.1 -> 52.5 | Helped |
| Use PASS with a 37% data budget instead of full-data simMIM pre-training | simMIM · ImageNet-1K | finetune Top-1 accuracy (%): 81.5 -> 82.1 | Helped |
| Use basic sample selection rather than full-data MAE pre-training | MAE · ImageNet-1K | finetune Top-1 accuracy (%): 82.2 -> 81.5 | Hurt |
| Use PASS instead of MAE-BSS | MAE · MS-COCO val2017 | APb: 38.46 -> 42.25 | Helped |
| Use PASS instead of MAE-BSS | MAE · ADE20K | mIoU: 39.52 -> 42.28 | Helped |
| Add Weighted Selection Score to DTMP (ablation) | PASS · ADE20K | mIoU: 41.83 -> 42.28 | Helped |
| Add Weighted Selection Score to DTMP (ablation) | PASS · ADE20K | aAcc: 80.87 -> 80.88 | No clear effect |
| Add Weighted Selection Score to DTMP (ablation) | PASS · ADE20K | mAcc: 52.20 -> 52.54 | Helped |
| Train the mask predictor dynamically before selection stages rather than statically for 20 epochs (ablation) | MAE + mask predictor · ImageNet-1K | finetune Top-1 accuracy (%): 82.49 -> 82.71 | Helped |
| Train the mask predictor dynamically before selection stages rather than statically for 100 epochs (ablation) | MAE + mask predictor · ImageNet-1K | finetune Top-1 accuracy (%): 82.54 -> 82.71 | Helped |
| Train the mask predictor dynamically before selection stages rather than statically for 200 epochs (ablation) | MAE + mask predictor · ImageNet-1K | finetune Top-1 accuracy (%): 82.75 -> 82.71 | No clear effect |
Surprising findings
- Basic sample selection with random masking degrades MIM performance noticeably below full-data MAE (e.g., 82.2 to 81.5 on ImageNet-1K finetune for MAE-BSS).
Limitations
- Experiments mainly use ViT-B/16 due to limited computation resources.
- Mask predictor adds extra training cost and hyperparameters (tau, gamma, mask ratio) need tuning.
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet-1K CIFAR-10 CIFAR-100 STL-10 MS-COCO ADE20K
Metrics: top-1 accuracy linear probing accuracy APb APm mIoU aAcc mAcc pre-training time speedup
Baselines: MAE MAE-BSS simMIM simMIM-BSS Random Init
Builds on: MAE SimMIM Coreset selection (Killamsetty et al. 2021) loss-based sample selection (Feldman 2020)
Closest papers by meaning
- From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modeling (CVPR 2025 · similarity 0.86)
- Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking (NeurIPS 2025 · similarity 0.85)
- Effective and Efficient Masked Image Generation Models (ICML 2025 · similarity 0.85)
- Enhanced Sample Selection with Confidence Tracking: Identifying Correctly Labeled Yet Hard-to-Learn Samples in Noisy Data (AAAI 2025 · similarity 0.85)
- Pre-training with Random Orthogonal Projection Image Modeling (ICLR 2024 · similarity 0.84)
- Boundary Matters: A Bi-Level Active Finetuning Method (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.