Research map / Architectures and efficiency / Attention and transformers
ELSA: Exploiting Layer-wise N:M Sparsity for Vision Transformer Acceleration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ELSA, a weight-sharing supernet framework that searches layer-wise N:M sparse configurations for vision transformers, yielding subnets with lower FLOPs and small ImageNet accuracy loss without per-config fine-tuning.
Problem
Choosing layer-wise N:M sparsity configurations for vision transformers on accelerators that support mixed N:M sparsity, since ViT blocks have equal parameter counts and per-config fine-tuning is expensive.
Approach
Exploit the subset-superset relation among N:M levels so all sparsity choices share one dense weight matrix per linear layer (dynamic magnitude-based masking), train a single supernet with knowledge distillation and a computation-cost-stratified two-step sampler, filter overly sparse choices automatically, and search with evolutionary search.
Key results
- ELSA-DeiT-B-N:4 reaches 6.0G FLOPs (1.53x reduction vs uniform 2:4 at 9.2G) with 81.4% Top-1, versus dense DeiT-B 81.8% at 17.6G.
- ELSA-Swin-B-N:4 reaches 5.3G FLOPs at 82.8% Top-1 versus uniform 2:4 at 8.0G and 83.1%.
- Subnets inherited from the supernet lose only 0.1-0.2% accuracy relative to 150-epoch fine-tuned 2:4 subnets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use layer-wise N:4 sparsity instead of uniform 2:4 sparsity | DeiT-B · ImageNet-1K val | FLOPs: 9.2G -> 6.0G | Helped |
| Use layer-wise N:4 sparsity instead of uniform 2:4 sparsity | Swin-B · ImageNet-1K val | FLOPs: 8G -> 5.3G | Helped |
| Use layer-wise N:4 sparsity instead of uniform 2:4 sparsity | DeiT-B · ImageNet-1K val | Top-1 accuracy: 81.6% -> 81.4% | No clear effect |
| Use two-step sampling rather than uniform configuration sampling (ablation) | DeiT-S supernet · ImageNet-1K, around 1.6G FLOPs | Top-1 accuracy: nearly 1% | Helped |
| Add automatic choice filtering to two-step sampling (ablation) | DeiT-B N:8 supernet · ImageNet-1K val, 8.0G FLOPs | Top-1 accuracy: 80.9% -> 81.6% | Helped |
| Add automatic choice filtering to two-step sampling (ablation) | DeiT-B N:8 supernet · ImageNet-1K val, 7.0G FLOPs | Top-1 accuracy: 80.5% -> 81.4% | Helped |
| Add automatic choice filtering to two-step sampling (ablation) | DeiT-B N:8 supernet · ImageNet-1K val, 6.0G FLOPs | Top-1 accuracy: 80.1% -> 81.1% | Helped |
| Use the supernet rather than the dense model to estimate subnet accuracy (ablation) | DeiT-B N:4 search · ImageNet-1K val, 7.0G FLOPs | Top-1 accuracy: 81.0% -> 81.5% | Helped |
| Inherit 2:4 weights from ELSA rather than fine-tune from pretrained weights (ablation) | DeiT-S · ImageNet-1K val | Top-1 accuracy: 79.2% -> 79.1% | No clear effect |
| Inherit 2:4 weights from ELSA rather than fine-tune from pretrained weights (ablation) | DeiT-B · ImageNet-1K val | Top-1 accuracy: 81.8% -> 81.6% | No clear effect |
| Inherit 2:4 weights from ELSA rather than fine-tune from pretrained weights (ablation) | Swin-S · ImageNet-1K val | Top-1 accuracy: 82.9% -> 82.8% | No clear effect |
| Inherit 2:4 weights from ELSA rather than fine-tune from pretrained weights (ablation) | Swin-B · ImageNet-1K val | Top-1 accuracy: 83.3% -> 83.1% | No clear effect |
Surprising findings
- ELSA's searched configurations assign higher sparsity to early blocks and lower sparsity to the larger middle/deeper linear layers, the opposite of the trend found by DominoSearch and ER on Swin-B.
Limitations
- Speedups are theoretical FLOP-based estimates; no measured runtime on actual N:M hardware is reported.
- Evaluation limited to ImageNet-1k classification on DeiT, Swin, and one CNN (ConvNext-S).
Experimental setup
Datasets: ImageNet-1k
Benchmarks: ImageNet-1k
Metrics: Top-1 accuracy FLOPs
Baselines: Erdos-Renyi (N:M adapted) DominoSearch Uniform 2:4 sparsity ASP fine-tuning
Builds on: Once-for-all supernet paradigm ASP DominoSearch Erdos-Renyi sparsity SR-STE
Closest papers by meaning
- Chasing Sparsity in Vision Transformers: An End-to-End Exploration (NeurIPS 2021 · similarity 0.87)
- An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (ICLR 2025 · similarity 0.83)
- SparseFormer: Sparse Visual Recognition via Limited Latent Tokens (ICLR 2024 · similarity 0.82)
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.82)
- BHViT: Binarized Hybrid Vision Transformer (CVPR 2025 · similarity 0.82)
- UniPTS: A Unified Framework for Proficient Post-Training Sparsity (CVPR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.