Research map / Architectures and efficiency / Attention and transformers
Super Vision Transformer
Mingbao Lin, Mengzhao Chen, Yuxin Zhang, Chunhua Shen, Rongrong Ji, Liujuan Cao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train one ViT with multi-size patches and multiple keeping rates (SuperViT) | DeiT-S (14x14, keep 1.0) · ImageNet-1k val | Top-1 accuracy (%): 79.8 -> 80.6 | Improved |
| Apply SuperViT at 14x14 and keep 0.5 (half the FLOPs) | DeiT-S · ImageNet-1k val | Top-1 accuracy (%): 79.8 -> 80.0 | Improved |
| Apply SuperViT to larger LV-ViT-S backbone | LV-ViT-S (14x14, keep 1.0) · ImageNet-1k val | Top-1 accuracy (%): 83.3 -> 83.5 | Improved |
| Use SuperViT's 8x8 keep 0.5 subnet instead of DeiT-T | DeiT-T · ImageNet-1k val | Top-1 accuracy (%): 72.2 -> 73.9 | Improved |
| Use SuperViT instead of EViT token dropping at same FLOPs | DeiT-S (3.0G FLOPs) · ImageNet-1k val | Top-1 accuracy (%): 79.5 -> 80.5 | Improved |
| Train each subnet individually from scratch vs shared SuperViT training (adding SuperViT) (ablation) | DeiT-S · ImageNet-1k val | Top-1 accuracy (%): 78.0% -> 80.0% | Improved |
| Add SuperViT training with only four subnets per iteration vs training all 12 (ablation) | DeiT-S · ImageNet-1k | training days: 8.8 -> 4.1 days | Improved |
| Fine-tune from SuperViT instead of DeiT-S on CIFAR-100 | DeiT-S · CIFAR-100 | Top-1 accuracy (%): 88.55 -> 89.33 | Improved |
| Use SuperViT pretrained backbone for UperNet segmentation | DeiT-S · ADE20K | mIoU: 42.96 -> 45.32 | Improved |
| Drop only low-attended tokens instead of also dropping 20% high-attended tokens (ablation) | SuperViT DeiT-S 14x14 keep 0.5 · ImageNet-1k val | Top-1 accuracy (%): 78.9 -> 80.0 | Improved |
| Drop tokens at blocks 4/7/10 instead of 4/6/8 (ablation) | SuperViT DeiT-S 8x8 keep 1.0 · ImageNet-1k val | Top-1 accuracy (%): 74.9 -> 75.8 | Improved |
| Drop tokens at blocks 4/7/10 instead of 3/6/9 (ablation) | SuperViT DeiT-S 8x8 keep 1.0 · ImageNet-1k val | Top-1 accuracy (%): 75.5 -> 75.8 | Improved |
Closest papers by meaning
- A Survey on Efficient Vision Transformers: Algorithms, Techniques, and Performance Benchmarking (TPAMI 2024 · similarity 0.90)
- ViTAS: Vision Transformer Architecture Search (ECCV 2022 · similarity 0.89)
- AutoFormer: Searching Transformers for Visual Recognition (ICCV 2021 · similarity 0.89)
- AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (CVPR 2022 · similarity 0.89)
- MiniViT: Compressing Vision Transformers With Weight Multiplexing (CVPR 2022 · similarity 0.89)
- Auto-scaling Vision Transformers without Training (ICLR 2022 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
