AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

GOHSP: A Unified Framework of Graph and Optimization-Based Heterogeneous Structured Pruning for Vision Transformer

Miao Yin, Burak Uzkent, Yilin Shen, Hongxia Jin, Bo Yuan

AAAI 2023 · Architectures and efficiency › Attention and transformers · approach: Vision transformer pruning · 11 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use graph-ranked head importance as a soft mask and jointly optimize heterogeneous structured sparsity rather than pruning heads and columns directlyViT-Small · CIFAR-10, 40% sparsityTop-1 accuracy: 96.93 -> 97.89Improved
Use GOHSP structured pruning instead of SSPDeiT-Tiny · ImageNet, 30% sparsityTop-1 accuracy: +1.65%Improved
Use GOHSP structured pruning instead of S²ViTEDeiT-Small · ImageNet, 40% sparsityTop-1 accuracy: +0.76%Improved
Compress ViT-Small with GOHSP at 40% sparsityViT-Small · CIFAR-10Top-1 accuracy: 97.85 -> 97.89Improved
Increase GOHSP sparsity from 40% to 80%ViT-Small · CIFAR-10Top-1 accuracy: 97.89 -> 97.40Hurt
Increase GOHSP sparsity from 40% to 50%DeiT-Small · ImageNetTop-1 accuracy: 79.98 -> 79.86Hurt
Use soft pruning instead of directly hard-pruning the lowest-ranked heads and lowest-norm MLP columns (ablation)ViT-Small · CIFAR-10Top-1 accuracyImproved
Estimate head ranking from a sampled batch rather than the full training dataset (ablation)GOHSP head-ranking algorithm · ViT-Small on CIFAR-10Head-ranking stabilityNo clear effect
Increase the penalty parameter ρ (ablation)GOHSP · ViT-Small on CIFAR-10Sparsity imposed during trainingImproved
Set the penalty parameter ρ too high (ablation)GOHSP · ViT-Small on CIFAR-10Top-1 accuracyHurt

Later papers that compared with this method

2 later accepted papers report GOHSP as a baseline in their results tables; 2 of them report a better number on most of the results they share with it.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy