Research map / Label-efficient and robust learning / Self-supervised and contrastive
On Data Scaling in Masked Image Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase pre-training data from IN1K (50%) to IN1K (100%) and train for 500K iterations | SwinV2-H with SimMIM · ImageNet-1K image classification | top-1 accuracy: less than 0.1 at 125K iterations; near 0.5 at 500K iterations | Helped |
| Increase pre-training data from IN1K (50%) to IN1K (100%) | SwinV2-S with SimMIM · ImageNet-1K image classification | top-1 accuracy | No clear effect |
| Increase pre-training data from IN1K (20%) to IN1K (100%) | SwinV2-H with SimMIM · ImageNet-1K image classification | top-1 accuracy: 84.4; 86.3 | Helped |
| Increase training length and data size for larger encoders | SwinV2-L with SimMIM · iNaturalist-18 | top-1 accuracy: 77.2 -> 81.9 | Helped |
| Use IN1K (100%) rather than IN1K (10%) for pre-training | SwinV2-L with SimMIM, 500K iterations · ADE20K | mIoU: 41.9 -> 51.1 | Helped |
| Train a large model longer on a small dataset | SwinV2-L with SimMIM on IN1K (10%) · ImageNet-1K image classification | top-1 accuracy: 83.7 | Mixed |
| Add a 47² sliding-window normalization to the reconstruction target (ablation) | SimMIM with SwinV2-L · ImageNet-1K image classification | top-1 accuracy: +0.3 | Helped |
| Use a 4-block decoder instead of a linear decoder (ablation) | SwinV2-B with SimMIM on IN1K (20%) · ImageNet-1K image classification | top-1 accuracy: 84.4 -> 84.4 | No clear effect |
| Use an 8-block decoder instead of a linear decoder (ablation) | SwinV2-B with SimMIM on IN1K (20%) · ImageNet-1K image classification | top-1 accuracy: 84.4 -> 84.5 | No clear effect |
| Sample 100 classes rather than 10% of images uniformly across all IN1K classes (ablation) | SwinV2-L with SimMIM, 500K iterations · ImageNet-1K image classification | top-1 accuracy: 83.5 -> 83.4 | No clear effect |
| Use validation loss as a proxy to select pre-trained models | SimMIM-pre-trained SwinV2 models · Multiple downstream tasks | Pearson correlation with fine-tuning performance: ImageNet-1K: -0.79 vs -0.64; iNaturalist: -0.54 vs -0.46; COCO detection: -0.81 vs -0.35; COCO segmentation: -0.86 vs -0 | Helped |
Closest papers by meaning
- From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modeling (CVPR 2025 · similarity 0.90)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.89)
- Morphing Tokens Draw Strong Masked Image Models (ICLR 2025 · similarity 0.88)
- Scalable Pre-training of Large Autoregressive Image Models (ICML 2024 · similarity 0.88)
- Pre-training with Random Orthogonal Projection Image Modeling (ICLR 2024 · similarity 0.87)
- Patch-Aware Sample Selection for Efficient Masked Image Modeling (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.