Research map / Architectures and efficiency / Attention and transformers
Global Vision Transformer Pruning With Hessian-Aware Saliency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Globally prune DeiT-B structures using Hessian-aware importance and latency regularization | DeiT-B · ImageNet-1K | Top-1 accuracy: 83.36 -> 83.29 | No clear effect |
| Apply post-training Ampere 2:4 sparsity to NViT-B | NViT-B · ImageNet-1K | parameter compression: 34M (2.57) -> 17M (5.14) | Helped |
| Prune DeiT-B to the NViT-S configuration | DeiT-B · ImageNet-1K | Top-1 accuracy: 81.20 -> 82.19 | Helped |
| Prune DeiT-B to the NViT-T configuration | DeiT-B · ImageNet-1K | Top-1 accuracy: 74.50 -> 76.21 | Helped |
| Redistribute block dimensions according to the ReViT rule | DeiT-S · ImageNet-1K | Top-1 accuracy: 81.01%* -> 81.22% | Helped |
| Redistribute block dimensions according to the ReViT rule | DeiT-T · ImageNet-1K | Top-1 accuracy: 72.84%* -> 74.20% | Helped |
| Add full-model distillation to CNN hard distillation (ablation) | Pruned NViT-T · ImageNet-1K, 50-epoch finetuning | Top-1 accuracy: 73.40 -> 73.55 | Helped |
| Use full-model distillation with cross-entropy instead of the proposed combined objective (ablation) | Pruned NViT-T · ImageNet-1K, 50-epoch finetuning | Top-1 accuracy: 73.55 -> 72.62 | Hurt |
| Remove both distillation losses and use cross-entropy only (ablation) | Pruned NViT-T · ImageNet-1K, 50-epoch finetuning | Top-1 accuracy: 73.55 -> 72.36 | Hurt |
| Prune only the embedding dimension by 50% (ablation) | DeiT-B · ImageNet-1K, single-component pruning | Top-1 accuracy: 82.13 -> 79.24 | Hurt |
| Prune only the MLP dimension by 50% (ablation) | DeiT-B · ImageNet-1K, single-component pruning | Top-1 accuracy: 82.13 | Hurt |
| Replace Hessian importance with group L2 magnitude (ablation) | DeiT-B · ImageNet-1K, DeiT-S latency target | Top-1 accuracy before finetuning: 76.59 -> 33.79 | Hurt |
Closest papers by meaning
- SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization (NeurIPS 2022 · similarity 0.92)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.89)
- RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers (ICML 2025 · similarity 0.89)
- MDP: Multidimensional Vision Model Pruning with Latency Constraint (CVPR 2025 · similarity 0.89)
- Elastic ViTs from Pretrained Models without Retraining (NeurIPS 2025 · similarity 0.88)
- Data-independent Module-aware Pruning for Hierarchical Vision Transformers (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.