Research map / Architectures and efficiency / Attention and transformers
TinyViT: Fast Pretraining Distillation for Small Vision Transformers
Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, Lu Yuan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add pretraining distillation on IN-21k (ablation) | DeiT-Ti · ImageNet-1k val | top-1 accuracy (%): 73.0 -> 74.4 | Improved |
| add pretraining distillation on IN-21k (ablation) | DeiT-S · ImageNet-1k val | top-1 accuracy (%): 80.5 -> 82.0 | Improved |
| add pretraining distillation on IN-21k (ablation) | Swin-T · ImageNet-1k val | top-1 accuracy (%): 81.9 -> 83.4 | Improved |
| pretrain on IN-21k without distillation instead of training on IN-1k (ablation) | Swin-T · ImageNet-1k val | top-1 accuracy (%): 81.2 -> 81.9 | Improved |
| add IN-21k pretraining without distillation (ablation) | TinyViT-21M · ImageNet-1k val | top-1 accuracy (%): 83.1 -> 83.8 | Improved |
| pretrain on cleaned IN-21k instead of original (ablation) | TinyViT-21M · ImageNet-1k val | top-1 accuracy (%): 83.8 -> 84.2 | Improved |
| distill from Florence instead of cleaning data (ablation) | TinyViT-21M · ImageNet-1k val | top-1 accuracy (%): 84.2 -> 84.8 | Improved |
| use CLIP-ViT-L/14 teacher instead of BEiT-L (ablation) | TinyViT-21M · ImageNet-1k val | top-1 accuracy (%): 84.1 -> 84.8 | Improved |
| use Florence teacher instead of CLIP-ViT-L/14 (ablation) | TinyViT-21M · ImageNet-Real | top-1 accuracy (%): 88.9 -> 89.0 | No clear effect |
| remove ground-truth labels from distillation (Florence teacher) (ablation) | TinyViT-21M variant · ImageNet-1k val | top-1 accuracy (%): 84.2 -> 84.9 | Improved |
| remove ground-truth labels from distillation (CLIP teacher) (ablation) | TinyViT-21M variant · ImageNet-1k val | top-1 accuracy (%): 84.3 -> 84.5 | Improved |
| replace Swin-T backbone with TinyViT-21M | Cascade Mask R-CNN · COCO val | AP: 48.1 -> 49.6 | Improved |
Later papers that compared with this method
1 later accepted paper reports TinyViT-5M as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation (NeurIPS 2024 · better on 3 of 3 shared results)
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- LeViT-256 — LeViT: A Vision Transformer in ConvNet's Clothing for Faster Inference (ICCV 2021) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- MiniViT: Compressing Vision Transformers With Weight Multiplexing (CVPR 2022 · similarity 0.92)
- TinyMIM: An Empirical Study of Distilling MIM Pre-Trained Models (CVPR 2023 · similarity 0.91)
- An Efficient Hybrid Vision Transformer for TinyML Applications (ICCV 2025 · similarity 0.91)
- Auto-scaling Vision Transformers without Training (ICLR 2022 · similarity 0.90)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.88)
- Asymmetric Masked Distillation for Pre-Training Small Foundation Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
