AI Paper Atlas

Research map / Detection and segmentation / CNN features

MST: Masked Self-Supervised Transformer for Visual Representation

Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, Jinqiao Wang

NeurIPS 2021 · Detection and segmentation › CNN features · approach: Plain and high-resolution ViTs · 29 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add random token masking (ablation)MST (DeiT-S, 100 epochs) · ImageNet linear probetop-1 accuracy (%): 73.1 -> 63.2Hurt
add attention-guided token masking (ablation)MST (DeiT-S, 100 epochs) · ImageNet linear probetop-1 accuracy (%): 73.1 -> 73.7Improved
choose mask candidates from lowest 1/8 attention tokens instead of all tokens (ablation)MST attention-guided mask · ImageNet linear probetop-1 accuracy (%): 63.2 -> 73.6Improved
use BN in projection head with MST update rule (ablation)MST · ImageNet linear probetop-1 accuracy (%): 73.1 -> 73.9Improved
use BN in projection head (ablation)baseline (no MST) · ImageNet linear probetop-1 accuracy (%): 72.4 -> 71.6Hurt
add global image decoder with original-image reconstruction (vs masked tokens only) with attention mask (ablation)MST · ImageNet linear probetop-1 accuracy (%): 71.9 -> 73.5Improved
use MST mask replacement instead of BERT-style replacement (ablation)MST · ImageNet linear probetop-1 accuracy (%): 73.5 -> 73.9Improved
vary restoration loss weight lambda2 (0.2 to 0.8) (ablation)MST · ImageNet linear probetop-1 accuracy (%): 73.7, 73.5, 73.9, 73.6No clear effect
train 300 epochs instead of 100MST (DeiT-S) · ImageNet linear probetop-1 accuracy (%): 75.0 -> 76.9Improved
use k=200 instead of k=10 in k-NN (ablation)MST · ImageNet k-NNtop-1 accuracy (%): 75.0 -> 71.8Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy