AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Training Vision Transformers with Only 2040 Images

Yun-Hao Cao, Hao Yu, Jianxin Wu

ECCV 2022 · Architectures and efficiency › Attention and transformers · approach: Hybrid and interpretable ViT designs · 71 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add label smoothing (ablation)InsDis (IDMM) · Flowers (from scratch)accuracy (%): 69.6 -> 70.4Improved
add small resolution (112 pretraining) (ablation)InsDis (IDMM) · Flowers (from scratch)accuracy (%): 70.4 -> 73.1Improved
add multi-crop (ablation)InsDis (IDMM) · Flowers (from scratch)accuracy (%): 73.1 -> 76.9Improved
add CutMix (ablation)InsDis (IDMM) · Flowers (from scratch)accuracy (%): 76.9 -> 79.9Improved
add projection MLP head (ablation)IDMM · Flowers (SIN-10k pretrain)accuracy (%): 86.6 -> 85.2Hurt
add projection MLP head (ablation)SimCLR · Flowers (SIN-10k pretrain)accuracy (%): 82.2 -> 83.3Improved
replace non-parametric contrastive pretraining with IDMMDeiT-Tiny · Flowers (200 ep fine-tune)accuracy (%): 72.3 -> 79.9Improved
add IDMM pretraining versus random initDeiT-Tiny · Flowers (from scratch, 800 ep)accuracy (%): 67.8 -> 83.4Improved
fine-tune at higher resolution 224->448 for 100 more epochsIDMM PVTv2-B3 · Flowersaccuracy (%): 95.9 -> 96.7Improved
pretrain on SIN-10k with IDMM before ImageNet trainingPVTv2-B0 · ImageNet top-1 (100 ep)accuracy (%): 68.6 -> 69.5Improved
pretrain on SIN-10k with IDMM before ImageNet trainingDeiT-Tiny · ImageNet top-1 (300 ep)accuracy (%): 72.2 -> 72.9Improved
sample SIN-total 10k without labels instead of balanced SIN-10k (ablation)IDMM PVTv2-B0 · ImageNet top-1 (100 ep)accuracy (%): 69.5 -> 69.5No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy