Research map / Architectures and efficiency / Attention and transformers
Chasing Sparsity in Vision Transformers: An End-to-End Exploration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| dynamic sparse training (SViTE) instead of OMP/GMP/TP pruning | DeiT-Small 50% sparsity · ImageNet-1K val | top-1 accuracy: 76.88 -> 79.72 | Helped |
| large sparse ViT instead of small dense ViT with similar parameters | DeiT-Base · ImageNet-1K val | top-1 accuracy: 78.59 -> 81.51 | Helped |
| structured S2ViTE prune-and-grow instead of SSP | DeiT-Small 40% structured sparsity · ImageNet-1K val | top-1 accuracy: 77.74 -> 79.22 | Helped |
| 40% structured sparsity via S2ViTE | DeiT-Base · ImageNet-1K val | top-1 accuracy: 81.80 -> 82.22 | Helped |
| add learnable token selector, 5% tokens dropped (ablation) | SViTE-Small 50% sparsity · ImageNet-1K val | top-1 accuracy: 79.72 -> 80.18 | Helped |
| drop 30% of tokens (ablation) | SViTE+-Small 50% sparsity · ImageNet-1K val | top-1 accuracy: 79.72 -> 77.90 | Hurt |
| add token selector to structured sparsity (ablation) | S2ViTE-Small 40% · ImageNet-1K val | top-1 accuracy: 79.22 -> 78.44 | Hurt |
| add SViTE weight sparsity on top of token selector only (ablation) | DeiT-Small with 10% data sparsity · ImageNet-1K val | top-1 accuracy: 78.67 -> 79.91 | Helped |
| update interval 20000 versus larger or smaller (ablation) | SViTE-Tiny 50% sparsity · ImageNet-1K val | top-1 accuracy | Helped |
| batch size 512 versus other options (ablation) | SViTE-Tiny 50% sparsity · ImageNet-1K val | top-1 accuracy | Helped |
| train dense baseline 600 instead of 300 epochs | DeiT-Base · ImageNet-1K val | top-1 accuracy: 81.80 -> 82.01 | Helped |
| apply SViTE to TNT-S backbone | TNT-S · ImageNet-1K val | top-1 accuracy: 81.50 -> 81.63 | Helped |
Closest papers by meaning
- Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training (AAAI 2023 · similarity 0.94)
- SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer (CVPR 2023 · similarity 0.92)
- When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations (ICLR 2022 · similarity 0.91)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.90)
- RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers (ICML 2025 · similarity 0.89)
- Patch Rebirth: Fast and Transferable Model Inversion of Vision Transformers (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.