Research map / Architectures and efficiency / Attention and transformers
An Experimental Study on Exploring Strong Lightweight Vision Transformers via Masked Image Modeling Pre-training
Jin Gang Gao, Shubo Lin, Shaoru Wang, Yutong Kou, Zeming Li, Liang Li, Congxuan Zhang, Xiaoqin Zhang, Yizheng Wang, Weiming Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add attention-map distillation from MAE ViT-Base on the last layer during MAE pre-training (ablation) | MAE ViT-Tiny · ADE20K | mIoU: 34.5 -> 42.9 | Improved |
| add attention-map distillation from MAE ViT-Base during MAE pre-training (ablation) | MAE ViT-Tiny · Pets | top-1 accuracy: 76.5 -> 89.1 | Improved |
| add attention-map distillation during MAE pre-training (ablation) | MAE ViT-Tiny · ImageNet-1K val | top-1 accuracy: 78.0 -> 78.3 | Improved |
| decouple distillation from reconstruction by connecting the decoder to layer 8 instead of 12 (D2-MAE) (ablation) | D-MAE ViT-Tiny · Pets | top-1 accuracy: 89.1 -> 90.4 | Improved |
| decouple distillation by connecting the decoder to layer 8 instead of 12 (ablation) | D-MAE ViT-Tiny · COCO detection | box AP: 42.0 -> 42.5 | Improved |
| add MAE reconstruction loss to attention distillation (ablation) | D2-MAE ViT-Tiny · Pets | top-1 accuracy: 84.5 -> 90.4 | Improved |
| add MAE reconstruction loss to attention distillation (ablation) | D2-MAE ViT-Tiny · COCO detection | box AP: 39.6 -> 42.5 | Improved |
| pre-train on IN21K instead of IN1K (ablation) | MAE ViT-Tiny · ImageNet-1K val | top-1 accuracy: 78.0 -> 78.0 | No clear effect |
| pre-train on IN21K instead of IN1K (ablation) | MoCo-v3 ViT-Tiny · ImageNet-1K val | top-1 accuracy: 76.8 -> 76.9 | No clear effect |
| reduce pre-training data to 1% of IN1K (ablation) | MoCo-v3 ViT-Tiny · ImageNet-1K val | top-1 accuracy: 76.8 -> 76.2 | Hurt |
| reduce pre-training data to 1% of IN1K (ablation) | MAE ViT-Tiny · ImageNet-1K val | top-1 accuracy: 78.0 -> 77.9 | No clear effect |
| use global average pooling instead of class token for fine-tuning (ablation) | MoCo-v3 ViT-Tiny · ImageNet-1K val | top-1 accuracy: 73.7 -> 76.8 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- SimMIM — SimMIM: A Simple Framework for Masked Image Modeling (CVPR 2022) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- A Closer Look at Self-Supervised Lightweight Vision Transformers (ICML 2023 · similarity 0.92)
- TinyMIM: An Empirical Study of Distilling MIM Pre-Trained Models (CVPR 2023 · similarity 0.89)
- Closest Neighbors are Harmful for Lightweight Masked Auto-encoders (CVPR 2025 · similarity 0.87)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.87)
- Morphing Tokens Draw Strong Masked Image Models (ICLR 2025 · similarity 0.87)
- A Theoretical Analysis of Self-Supervised Learning for Vision Transformers (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
