Research map / Architectures and efficiency / Attention and transformers
Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks
Yongming Rao, Zuyan Liu, Wenliang Zhao, Jie Zhou, Jiwen Lu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use dynamic (input-dependent) token sparsification instead of structural downsampling (ablation) | DynamicViT (DeiT-S) · ImageNet val | top-1 accuracy (%): 78.2 -> 79.3 | Improved |
| use dynamic rather than static sparsification (ablation) | DynamicViT (DeiT-S) · ImageNet val | top-1 accuracy (%): 73.4 -> 79.3 | Improved |
| use learned prediction module instead of random token removal (ablation) | DynamicViT (DeiT-S) · ImageNet val | top-1 accuracy (%): 77.5 -> 79.3 | Improved |
| use learned prediction module instead of class-attention-based removal (ablation) | DynamicViT (DeiT-S) · ImageNet val | top-1 accuracy (%): 78.1 -> 79.3 | Improved |
| use three progressive stages instead of a single stage (ablation) | DynamicViT (DeiT-S) · ImageNet val | top-1 accuracy (%): 77.4 -> 79.3 | Improved |
| use three stages instead of two (ablation) | DynamicViT (DeiT-S) · ImageNet val | top-1 accuracy (%): 79.2 -> 79.3 | Improved |
| add distillation and KL losses (ablation) | DynamicViT-S/0.7 (LV-ViT-S) · ImageNet val | top-1 accuracy (%): 82.5 -> 83.0 | Improved |
| add distillation and KL losses (ablation) | DynamicViT-S/0.7 (DeiT-S) · ImageNet val | top-1 accuracy (%): 79.2 -> 79.3 | Improved |
| use linear projection fast path instead of zero mask (ablation) | DynamicCNN-S/0.9 · ImageNet val | top-1 accuracy (%): 53.2 -> 83.1 | Improved |
| use linear projection instead of learnable mask (ablation) | DynamicCNN-S/0.9 · ImageNet val | top-1 accuracy (%): 55.5 -> 83.1 | Improved |
| use linear projection instead of bottleneck MLP fast path (ablation) | DynamicCNN-S/0.9 · ImageNet val | top-1 accuracy (%): 82.9 -> 83.1 | Improved |
| sparsify only stage 3 instead of also stage 4 (ablation) | DynamicCNN-T · ImageNet val | top-1 accuracy (%): 79.7 -> 81.6 | Improved |
Closest papers by meaning
- DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification (NeurIPS 2021 · similarity 0.92)
- SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer (CVPR 2023 · similarity 0.89)
- Chasing Sparsity in Vision Transformers: An End-to-End Exploration (NeurIPS 2021 · similarity 0.89)
- Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers (ICML 2025 · similarity 0.88)
- An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (ICLR 2025 · similarity 0.87)
- SparseDiT: Token Sparsification for Efficient Diffusion Transformer (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
