Research map / Architectures and efficiency / Pruning and efficient tuning
Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Zero-TPrune prunes tokens in pre-trained vision Transformers without any fine-tuning, using a Weighted PageRank over the attention graph for importance and importance-guided similarity pruning.
Problem
Token pruning for Transformers usually needs expensive fine-tuning after pruning, which is impractical for edge deployment and for pruning large models under many configurations.
Approach
Treats the attention matrix as an adjacency matrix of a directed attention graph and ranks tokens with a Weighted PageRank (WPR) iteration, then combines this importance distribution with key-based cosine similarity pruning via importance-guided bipartite partitioning, with EIR head aggregation and variance-based head filtering.
Key results
- On DeiT-S without fine-tuning, Zero-TPrune cuts GFLOPS by 34.7% and raises throughput by 45.3% with 0.4% top-1 accuracy loss (79.4% vs 79.8%).
- Compared with fine-tuning-free methods, Zero-TPrune reduces accuracy loss by up to 49% on medium-sized models such as AugReg and LV-ViT-S (AugReg 80.22% vs ToMe 79.30% at ~2.8 GFLOPS).
- Without fine-tuning, Zero-TPrune is about 1% better than DynamicViT and A-ViT with random drop, and close to those methods with fine-tuning (0.1% loss relative to the best at 3.5 GFLOPS on DeiT-S).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace random token dropping with WPR importance scoring (ablation) | DeiT-S with Zero-TPrune · ImageNet | Acc@1: 76.8% (-3.0%) -> 78.6% (+1.8%) | Helped |
| add EIR aggregation to WPR (ablation) | DeiT-S with Zero-TPrune · ImageNet | Acc@1: 78.6% (+1.8%) -> 78.8% (+0.2%) | Helped |
| add VHF to WPR+EIR (ablation) | DeiT-S with Zero-TPrune · ImageNet | Acc@1: 78.8% (+0.2%) -> 78.9% (+0.1%) | Helped |
| add similarity-based S-stage to the I-stage (ablation) | DeiT-S with Zero-TPrune · ImageNet | Acc@1: 78.9% (+0.1%) -> 79.4% (+0.5%) | Helped |
| apply Zero-TPrune rather than ATS | DeiT-S · ImageNet, 224px | Acc@top1: 79.2% (-0.6%) -> 79.4% (-0.4%) | Helped |
| apply Zero-TPrune rather than ToMe | DeiT-S · ImageNet, 224px | Acc@top1: 78.9% (-0.9%) -> 79.4% (-0.4%) | Helped |
| apply Zero-TPrune rather than ATS or ToMe | AugReg · ImageNet | Acc@top1: 79.21% -> 80.22%; 79.30% -> 80.22% | Helped |
| apply Zero-TPrune rather than ATS or ToMe | LV-ViT-S · ImageNet | Acc@top1: 80.4% -> 81.5%; 79.8% -> 81.5% | Helped |
| apply Zero-TPrune rather than ATS or ToMe | MAE · ImageNet | Acc@top1: 82.07 -> 82.93; 82.69 -> 82.93 | Helped |
| apply Zero-TPrune rather than ATS or ToMe | SWAG · ImageNet, 384px | Acc@top1: 84.21 -> 85.17; 85.09 -> 85.17 | Helped |
| use Zero-TPrune without fine-tuning after pruning instead of fine-tuned DynamicViT or A-ViT | DeiT-S · ImageNet | Acc@1: 0.1% accuracy loss relative to the best | No clear effect |
Surprising findings
- Aggressively pruning large models does not beat baselines; a smaller pretrained model is often the better choice.
Limitations
- Evaluated only on image classification; applicability to other tasks is untested.
- Aggressively pruning large models does not outperform baselines.
- Accuracy loss remains (0.4% on DeiT-S at 34.7% FLOPs reduction) compared with the unpruned model.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet ImageNet downstream transfer datasets
Metrics: Acc@1 GFLOPS throughput (img/s)
Baselines: DynamicViT A-ViT ATS ToMe
Builds on: PageRank ToMe ATS DynamicViT
Closest papers by meaning
- CORP: Closed-Form One-shot Representation-Preserving Structured Pruning for Transformers (NeurIPS 2026 · similarity 0.90)
- Exploring Token Pruning in Vision State Space Models (NeurIPS 2024 · similarity 0.89)
- Collaborative Multi-Mode Pruning for Vision-Language Models (CVPR 2026 · similarity 0.88)
- UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer (AAAI 2024 · similarity 0.87)
- MDP: Multidimensional Vision Model Pruning with Latency Constraint (CVPR 2025 · similarity 0.87)
- The Need for Speed: Pruning Transformers with One Recipe (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.