Research map / Architectures and efficiency / Attention and transformers
Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TPS token squeezing after pruning | DeiT token pruning · ImageNet1K, aggressive pruning | Top-1 accuracy: 1%-6% | Helped |
| Add TPS token squeezing | PVT-T · ImageNet1K | Top-1 accuracy: 75.1 -> 75.2 | Helped |
| Add TPS token squeezing | PVT-S · ImageNet1K | Top-1 accuracy: 79.8 -> 79.2 | Hurt |
| Add TPS token squeezing | CvT-13 · ImageNet1K | Top-1 accuracy: 81.6 -> 80.8 | Hurt |
| Add TPS token squeezing | CvT-21 · ImageNet1K | Top-1 accuracy: 82.5 -> 80.9 | Hurt |
| Use full token embeddings rather than content-only features for matching (ablation) | dTPS · ImageNet1K, DeiT-T; keep ratio 0.7 | Top-1 accuracy: 71.73 -> 71.90 | Helped |
| Use full token embeddings rather than position-only features for matching (ablation) | dTPS · ImageNet1K, DeiT-T; keep ratio 0.7 | Top-1 accuracy: 70.92 -> 71.90 | Helped |
| Compute cosine similarity from current token features instead of reusing previous attention (ablation) | dTPS · ImageNet1K, DeiT-T; keep ratio 0.7 | Top-1 accuracy: 71.35 -> 71.90 | Helped |
| Compute cosine similarity from current token features instead of reusing previous attention (ablation) | eTPS · ImageNet1K, DeiT-T; keep ratio 0.7 | Top-1 accuracy: 71.67 -> 72.26 | Helped |
| Use many-to-one matching from every pruned token to its nearest reserved host rather than one-to-one matching (ablation) | dTPS · ImageNet1K, DeiT-T; keep ratio 0.7 | Accuracy: 69.02 -> 71.90 | Helped |
| Use similarity-weighted fusing rather than average fusing (ablation) | dTPS · ImageNet1K, DeiT-T; random token squeezing | Accuracy: 65.173 -> 65.56 | Helped |
| Train dTPS for 100 rather than 30 epochs (ablation) | DeiT-T · ImageNet1K | Top-1 accuracy: 71.7 -> 72.4 | Helped |
Closest papers by meaning
- Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers (CVPR 2024 · similarity 0.91)
- Exploring Token Pruning in Vision State Space Models (NeurIPS 2024 · similarity 0.90)
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.90)
- Token Cropr: Faster ViTs for Quite a Few Tasks (CVPR 2025 · similarity 0.89)
- Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers (CVPR 2024 · similarity 0.89)
- Collaborative Multi-Mode Pruning for Vision-Language Models (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.