Research map / Architectures and efficiency / Attention and transformers
A Survey on Vision Transformer
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, Zhaohui Yang, Yiman Zhang and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add token-based distillation from a CNN teacher | DeiT-B · ImageNet-1k val | top-1 accuracy (%): 83.1 -> 84.40 | Improved |
| pre-train on JFT-300M instead of training on ImageNet only | ViT · ImageNet | top-1 accuracy (%): 88.36 | Improved |
| replace convolutions with deformable attention in DETR | Deformable DETR · COCO 2017 val | AP: 42.0 -> 46.2 | Improved |
| add convolution to transformer (CNN + Transformer hybrids) | vision backbones · ImageNet-1k val | top-1 accuracy (%): 83.3 -> 84.5 | Improved |
| use pyramid architecture with local windows (Swin-B) vs plain DeiT-B | Swin-B · ImageNet-1k val | top-1 accuracy (%): 81.8 -> 83.3 | Improved |
| scale VOLO from D1 to D5 | VOLO · ImageNet-1k val | top-1 accuracy (%): 84.2 -> 86.1 | Improved |
| train ViT on mid-sized ImageNet instead of large-scale data | ViT · ImageNet | top-1 accuracy (%) | Hurt |
Closest papers by meaning
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ICLR 2021 · similarity 0.91)
- Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attention (AAAI 2022 · similarity 0.90)
- A Survey on Efficient Vision Transformers: Algorithms, Techniques, and Performance Benchmarking (TPAMI 2024 · similarity 0.90)
- Three Things Everyone Should Know about Vision Transformers (ECCV 2022 · similarity 0.90)
- Lite Vision Transformer With Enhanced Self-Attention (CVPR 2022 · similarity 0.90)
- What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
