Research map / Architectures and efficiency / Attention and transformers
What to Hide from Your Students: Attention-Guided Masked Image Modeling
Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, Nikos Komodakis
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace random block-wise masking with attention-guided masking of most attended tokens (AttMask-High) (ablation) | iBOT ViT-S, 20% ImageNet · ImageNet-1k k-NN | top-1 accuracy: 46.7 -> 49.7 | Improved |
| replace random block-wise masking with AttMask-High (ablation) | iBOT ViT-S, 20% ImageNet · ImageNet-1k linear probing | top-1 accuracy: 56.4 -> 57.9 | Improved |
| mask least attended tokens (AttMask-Low) instead of random block-wise (ablation) | iBOT ViT-S, 20% ImageNet · ImageNet-1k k-NN | top-1 accuracy: 46.7 -> 44.0 | Hurt |
| use aggressive 75% random masking instead of block-wise 10-50% (ablation) | iBOT ViT-S, 20% ImageNet · ImageNet-1k linear probing | top-1 accuracy: 56.4 -> 55.5 | Hurt |
| use random patch masking (10-50%) instead of block-wise (ablation) | iBOT ViT-S, 20% ImageNet · ImageNet-1k k-NN | top-1 accuracy: 46.7 -> 47.8 | Improved |
| reveal a few most attended tokens as hints (AttMask-Hint) rather than hiding all (AttMask-High) (ablation) | iBOT ViT-S, 20% ImageNet · ImageNet-1k k-NN | top-1 accuracy: 49.7 -> 49.5 | No clear effect |
| add AttMask-High masking to DINO (no MIM loss) (ablation) | DINO ViT-S, 20% ImageNet · ImageNet-1k k-NN | top-1 accuracy: 43.0 -> 43.5 | Improved |
| use AttMask-Low masking in DINO (ablation) | DINO ViT-S, 20% ImageNet · ImageNet-1k k-NN | top-1 accuracy: 43.0 -> 42.7 | Hurt |
| use AttMask-High with less training data (5% of ImageNet) (ablation) | iBOT ViT-S · ImageNet-1k k-NN | top-1 accuracy: 15.7 -> 17.5 | Improved |
| train on 100% ImageNet for 300 epochs with AttMask (ablation) | iBOT ViT-S · ImageNet-1k k-NN | top-1 accuracy: 74.6 -> 75.0 | Improved |
| add AttMask-High to iBOT with few examples per class (nu=1) (ablation) | iBOT ViT-S, 100 ep · ImageNet-1k k-NN few-shot | top-1 accuracy: 32.9 -> 37.1 | Improved |
| use masking probability 0.5 rather than 0 (no masking) for attention-guided masking (ablation) | iBOT+AttMask · ImageNet-1k k-NN | top-1 accuracy: 43.4 -> 49.4 | Improved |
Closest papers by meaning
- Emerging Property of Masked Token for Effective Pre-training (ECCV 2024 · similarity 0.91)
- Learning with Unmasked Tokens Drives Stronger Vision Learners (ECCV 2024 · similarity 0.91)
- Morphing Tokens Draw Strong Masked Image Models (ICLR 2025 · similarity 0.91)
- On the Role of Discrete Tokenization in Visual Representation Learning (ICLR 2024 · similarity 0.89)
- From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modeling (CVPR 2025 · similarity 0.89)
- Asymmetric Masked Distillation for Pre-Training Small Foundation Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
