Research map / Architectures and efficiency / Attention and transformers
SNP: Structured Neuron-level Pruning to Preserve Attention Scores
KyungHwan Shim, Jaewoong Yun, Shinkook Choi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| prune query/key layers with SNP and value layers for inter-head redundancy on DeiT-Small | DeiT-Small · ImageNet-1K | Jetson Nano latency (ms): 99.32 -> 32.03 | Improved |
| use large pruned DeiT-Small with SNP instead of hand-crafted DeiT-Tiny | DeiT-Small · ImageNet-1K | Top-1 accuracy (%): 72.20 -> 73.32 | Improved |
| combine SNP with head pruning on DeiT-Base (ablation) | DeiT-Base · ImageNet-1K | RTX 3090 latency (ms): 72.96 -> 39.31 | Improved |
| combine traditional neuron-level masking with head pruning (ablation) | head pruning · ImageNet-1K | latency | No clear effect |
| compute SNP importance scores with more images (1 to 256) (ablation) | SNP · DeiT-Tiny, 70% pruning, no fine-tuning | Top-1 accuracy (%): 38.68 -> 42.70 | Improved |
| use SNP criterion instead of l2-norm, geometric median or Reverse for query/key pruning (ablation) | DeiT-Tiny · ImageNet-1K, no fine-tuning | Top-1 accuracy | Improved |
| apply SNP versus WDPruning on DeiT-Base | DeiT-Base · ImageNet-1K | GPU speedup: 1.18x -> 2.07x | Improved |
| apply SNP to DeiT-Tiny on GPU versus WDPruning | DeiT-Tiny · ImageNet-1K | GPU speedup: 1.19x -> 1.05x | Hurt |
| convert DeiT MSA split/reshape order for SNP (ablation) | DeiT MSA modules · RTX 3090 latency | latency: slower by 0.72 ms (Tiny), 0.43 ms (Base) | No clear effect |
Closest papers by meaning
- GOHSP: A Unified Framework of Graph and Optimization-Based Heterogeneous Structured Pruning for Vision Transformer (AAAI 2023 · similarity 0.88)
- Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.86)
- Data-independent Module-aware Pruning for Hierarchical Vision Transformers (ICLR 2024 · similarity 0.85)
- Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers (CVPR 2024 · similarity 0.85)
- MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention (NeurIPS 2025 · similarity 0.85)
- UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
