Research map / Architectures and efficiency / Attention and transformers
Training Vision Transformers with Only 2040 Images
Yun-Hao Cao, Hao Yu, Jianxin Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add label smoothing (ablation) | InsDis (IDMM) · Flowers (from scratch) | accuracy (%): 69.6 -> 70.4 | Improved |
| add small resolution (112 pretraining) (ablation) | InsDis (IDMM) · Flowers (from scratch) | accuracy (%): 70.4 -> 73.1 | Improved |
| add multi-crop (ablation) | InsDis (IDMM) · Flowers (from scratch) | accuracy (%): 73.1 -> 76.9 | Improved |
| add CutMix (ablation) | InsDis (IDMM) · Flowers (from scratch) | accuracy (%): 76.9 -> 79.9 | Improved |
| add projection MLP head (ablation) | IDMM · Flowers (SIN-10k pretrain) | accuracy (%): 86.6 -> 85.2 | Hurt |
| add projection MLP head (ablation) | SimCLR · Flowers (SIN-10k pretrain) | accuracy (%): 82.2 -> 83.3 | Improved |
| replace non-parametric contrastive pretraining with IDMM | DeiT-Tiny · Flowers (200 ep fine-tune) | accuracy (%): 72.3 -> 79.9 | Improved |
| add IDMM pretraining versus random init | DeiT-Tiny · Flowers (from scratch, 800 ep) | accuracy (%): 67.8 -> 83.4 | Improved |
| fine-tune at higher resolution 224->448 for 100 more epochs | IDMM PVTv2-B3 · Flowers | accuracy (%): 95.9 -> 96.7 | Improved |
| pretrain on SIN-10k with IDMM before ImageNet training | PVTv2-B0 · ImageNet top-1 (100 ep) | accuracy (%): 68.6 -> 69.5 | Improved |
| pretrain on SIN-10k with IDMM before ImageNet training | DeiT-Tiny · ImageNet top-1 (300 ep) | accuracy (%): 72.2 -> 72.9 | Improved |
| sample SIN-total 10k without labels instead of balanced SIN-10k (ablation) | IDMM PVTv2-B0 · ImageNet top-1 (100 ep) | accuracy (%): 69.5 -> 69.5 | No clear effect |
Closest papers by meaning
- DeiT III: Revenge of the ViT (ECCV 2022 · similarity 0.90)
- Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets (NeurIPS 2022 · similarity 0.90)
- Do Vision Transformers See Like Convolutional Neural Networks? (NeurIPS 2021 · similarity 0.90)
- Unified Visual Transformer Compression (ICLR 2022 · similarity 0.90)
- Efficient Training of Visual Transformers with Small Datasets (NeurIPS 2021 · similarity 0.90)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
