Research map / Architectures and efficiency / Attention and transformers

Data-independent Module-aware Pruning for Hierarchical Vision Transformers

ICLR 2024 · Architectures and efficiency › Attention and transformers · approach: Hybrid and interpretable ViT designs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DIMAP, a data-independent, module-aware weight-magnitude pruning method for hierarchical ViTs (Swin) that ranks weights by Frobenius distortion within modules, achieving large FLOPs reduction with little accuracy loss on ImageNet-1K.

Problem

Magnitude pruning of hierarchical vision transformers (Swin) compares local window attention weights globally and ignores distinct per-layer weight distributions, and input-dependent importance criteria are tied to the patch-merging process.

Approach

Groups layers into functional modules (QKV-M, PRJ-M, MLP-M, AUX-M) and ranks weights within a module by a data-independent importance score, the squared weight over the cumulative sum of squared weights, derived from bounding the Frobenius distortion of pruning.

Key results

• Swin-B-DIMAP3 removes 52.5% parameters and 52.7% FLOPs with a 0.20% top-1 drop (83.28%) and 0.07% top-5 drop.
• Swin-S-DIMAP2 removes 33.2% parameters and FLOPs with top-5 accuracy 0.03% above the unpruned Swin-S.
• Versus uniform magnitude pruning at 50% removal, DIMAP gains over 60 top-1 points on Swin-B, Swin-S and Swin-T.

Measured effects reported in the paper

Surprising findings

• Pruned Swin-B-DIMAP1 slightly exceeds the unpruned Swin-B in top-1 accuracy (83.52 vs 83.48).
• Large pruned models (e.g., Swin-B-DIMAP3) outperform smaller unpruned models (Swin-S) in accuracy with fewer parameters.

Limitations

• Weight pruning gives no realistic acceleration on GPUs, as the authors note.
• Evaluated only on ImageNet-1K classification; no downstream detection or segmentation results.
• Only Swin Transformer variants are tested, not other hierarchical ViTs.

Experimental setup

Datasets: ImageNet-1K

Benchmarks: ImageNet-1K

Metrics: top-1 accuracy top-5 accuracy parameter count FLOPs

Baselines: VTP UVC SViTE AutoFormer As-ViT X-Pruner WDPruning NViT-S uniform magnitude pruning

Builds on: magnitude pruning Swin Transformer UVC

Closest papers by meaning

• The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy (CVPR 2022 · similarity 0.89)
• Synergistic Patch Pruning for Vision Transformer: Unifying Intra- & Inter-Layer Patch Importance (ICLR 2024 · similarity 0.89)
• SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization (NeurIPS 2022 · similarity 0.88)
• HiViT: A Simpler and More Efficient Design of Hierarchical Vision Transformer (ICLR 2023 · similarity 0.88)
• MDP: Multidimensional Vision Model Pruning with Latency Constraint (CVPR 2025 · similarity 0.88)
• FasterViT: Fast Vision Transformers with Hierarchical Attention (ICLR 2024 · similarity 0.87)