Research map / Architectures and efficiency / Attention and transformers

Synergistic Patch Pruning for Vision Transformer: Unifying Intra- & Inter-Layer Patch Importance

ICLR 2024 · Architectures and efficiency › Attention and transformers · approach: Vision transformer pruning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Synergistic Patch Pruning (SPP) for ViTs that prunes image patches by unifying intra-layer and inter-layer patch importance to reduce computation.

Problem

ViTs are computationally expensive, and the attention mechanism that aggregates patch information into the [CLS] token makes traditional structured CNN pruning unsuitable.

Approach

Unifies intra-layer patch importance (contribution to the [CLS] token within a layer) with inter-layer patch importance (how patch importance evolves across layers) to decide which patches to prune, rather than pruning per-layer independently.

Closest papers by meaning

• FlexiViT: One Model for All Patch Sizes (CVPR 2023 · similarity 0.90)
• SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization (NeurIPS 2022 · similarity 0.90)
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration (NeurIPS 2021 · similarity 0.89)
• Data-independent Module-aware Pruning for Hierarchical Vision Transformers (ICLR 2024 · similarity 0.89)
• Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More (ICML 2025 · similarity 0.88)
• Probabilistic Conceptual Explainers: Trustworthy Conceptual Explanations for Vision Foundation Models (ICML 2024 · similarity 0.87)