Research map / Architectures and efficiency / Attention and transformers
Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space
Arnav Chavan, Zhiqiang Shen, Zhuang Liu, Zechun Liu, Kwang-Ting Cheng, Eric P. Xing
Check your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Search MHSA dimensions only instead of retaining all dimensions (ablation) | DeiT-S · ImageNet-1K | Top-1 accuracy: 79.90 -> 80.90 | Improved |
| Search MLP dimensions only instead of retaining all dimensions (ablation) | DeiT-S · ImageNet-1K | Top-1 accuracy: 79.90 -> 80.80 | Improved |
| Use joint MHSA and MLP dimension search at 70% budget (ablation) | DeiT-S · ImageNet-1K | Top-1 accuracy: 79.90 -> 80.03 | Improved |
| Reduce the joint MHSA and MLP budget from 60% to 50% (ablation) | DeiT-S ViT-Slim-PS · ImageNet-1K | Top-1 accuracy: 79.20 -> 77.94 | Hurt |
| Search MLP dimensions rather than MHSA dimensions at equal FLOPs (ablation) | DeiT-S · ImageNet-1K | Top-1 accuracy: MHSA 40%: 79.61; MLP 60%: 80.39 | Mixed |
| Add patch selection and reduce active patches from 100% to 80% (ablation) | DeiT-S ViT-Slim-JS, MHSA and MLP fixed at 70% · ImageNet-1K | Top-1 accuracy: 80.03 -> 79.91 | No clear effect |
| Reduce active patches from 100% to 70% (ablation) | DeiT-S ViT-Slim-JS, MHSA and MLP fixed at 70% · ImageNet-1K | Top-1 accuracy: 80.03 -> 79.72 | No clear effect |
| Reduce active patches from 100% to 60% (ablation) | DeiT-S ViT-Slim-JS, MHSA and MLP fixed at 70% · ImageNet-1K | Top-1 accuracy: 80.03 -> 79.51 | Hurt |
| Tune the MHSA and MLP sparsity weights to 2e-04 and 5e-05 (ablation) | DeiT-S ViT-Slim-PS at 60% budget · ImageNet-1K | final Top-1 accuracy: 79.10 -> 79.20 | Improved |
| Use joint MHSA and MLP search at 60% budget (ablation) | DeiT-S ViT-Slim-PS · ImageNet-1K | Top-1 accuracy: 79.90 -> 79.20 | Hurt |
| Use joint MHSA and MLP search at 50% budget (ablation) | DeiT-S ViT-Slim-PS · ImageNet-1K | Top-1 accuracy: 79.90 -> 77.94 | Hurt |
| Compress DeiT-B using ViT-Slim at 60% budget | DeiT-B · ImageNet-1K | Top-1 accuracy: 81.8 -> 82.4 | Improved |
Closest papers by meaning
- Self-Slimmed Vision Transformer (ECCV 2022 · similarity 0.90)
- Patch Slimming for Efficient Vision Transformers (CVPR 2022 · similarity 0.88)
- Unified Visual Transformer Compression (ICLR 2022 · similarity 0.88)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.86)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.86)
- Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compression (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
