Research map / Architectures and efficiency / Attention and transformers
Adaptive Pruning of Pretrained Transformer via Differential Inclusions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Prunes pretrained transformers (DeiT, Swin, CLIP, and LLMs) at any ratio from a single search run by generating a regularization solution path of column masks via a differential inclusion (mirror descent / Bregman inverse scale space).
Problem
Pruning pretrained transformers usually requires a separate pruning process for each target compression ratio, which is computationally costly.
Approach
Uses a differential inclusion (linearized Bregman / inverse scale space dynamics) on mask parameters with an l1 penalty and an auxiliary sparse variable, so one search run yields a solution path whose support sets give a family of pruned transformers at varying sparsity; paired-module (QK, VO, MLP) shared masks enable structural pruning.
Key results
- On DeiT-Small, the pruned model reaches 80.2% top-1 at 73.9% FLOPs (about 70.6% params), matching uncompressed 79.8% and exceeding ViT-Slim's 80.0% at similar size.
- On DeiT-Base, a 25.4%-parameter model reaches 78.1% top-1 while the 55.5%-param model gets 81.9%, above the 81.8% uncompressed baseline.
- On Llama2-7B at 50% sparsity, SPP reaches ARC-e 71.80 with C4 calibration, comparable to Wanda and RIA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPP instead of DessiLBI training from scratch (ablation) | DeiT-Small · ImageNet-1k | Top-1 accuracy: 78.9 -> 80.2 | Helped |
| Use SPP instead of DessiLBI training from scratch (ablation) | DeiT-Tiny · ImageNet-1k | Top-1 accuracy: 71.8 -> 72.3 | Helped |
| Use SPP instead of DessiLBI training from scratch (ablation) | Swin-Tiny · ImageNet-1k | Top-1 accuracy: 80.4 -> 80.6 | Helped |
| Prune DeiT-Small with SPP at the reported 29.5% parameter reduction | DeiT-Small · ImageNet-1k | Top-1 accuracy: 79.8 -> 80.2 | Helped |
| Prune DeiT-Tiny with SPP | DeiT-Tiny · ImageNet-1k | Top-1 accuracy: 72.2 -> 72.3 | Helped |
| Prune DeiT-Base with SPP at the reported compression levels | DeiT-Base · ImageNet-1k | Top-1 accuracy: 81.8 -> 81.9; 81.8 -> 81.2; 81.8 -> 78.1 | Mixed |
| Prune Swin-Tiny with SPP | Swin-Tiny · ImageNet-1k | Top-1 accuracy: 81.2 -> 80.6 | Hurt |
| Prune DeiT-Small with SPP | DeiT-Small · CIFAR-10 | Top-1 accuracy: 98.5 -> 98.8 | Helped |
| Prune CLIP-Large with SPP at the reported sparsity levels | CLIP-Large · COCO, image-to-text retrieval | R@1: 71.5 -> 73.7; 71.5 -> 70.3 | Mixed |
| Prune CLIP-Base with SPP at the reported sparsity levels | CLIP-Base · COCO, image-to-text retrieval | R@1: 52.5 -> 69.0; 52.5 -> 31.4 | Hurt |
| Prune Llama2-7B with SPP using C4 calibration | Llama2-7B · ARC-c | accuracy: 38.40 -> 38.57 | Helped |
| Prune OPT-6.7B with SPP using C4 calibration | OPT-6.7B · ARC-c | accuracy: 29.27 -> 28.75 | Hurt |
Surprising findings
- The pruned DeiT-Base (55.5% params) reports 81.9% top-1, slightly above the uncompressed 81.8% baseline.
Limitations
- LLM extension relies on combining with the RIA metric and only reaches comparable, not superior, results to Wanda/RIA
- Results on some tables mix different baselines and training budgets, making comparisons inconsistent
- Convergence theorem depends on Kurdyka-Łojasiewicz assumptions that are stated but not verified for transformers
Experimental setup
Datasets: ImageNet-1k COCO CIFAR-10 C4 Wikitext2
Benchmarks: ImageNet-1k COCO CIFAR-10 ARC-c ARC-e BoolQ RTE SST
Metrics: Top-1 accuracy Top-5 accuracy Recall@K FLOPS Params zero-shot accuracy
Baselines: ViT-Slim UPop WDPruning S2ViTE GOHSP X-Pruner OPTIN SCOP PoWER IA-RED DessiLBI RIA
Builds on: DessiLBI Linearized Bregman Iteration Mirror descent RIA Wanda
Closest papers by meaning
- Dense Structural Compression of Transformers via Gauge-Correct Channel Removal (NeurIPS 2026 · similarity 0.86)
- Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers (CVPR 2024 · similarity 0.85)
- TinyFusion: Diffusion Transformers Learned Shallow (CVPR 2025 · similarity 0.84)
- CORP: Closed-Form One-shot Representation-Preserving Structured Pruning for Transformers (NeurIPS 2026 · similarity 0.84)
- Unraveling the Gradient Descent Dynamics of Transformers (NeurIPS 2024 · similarity 0.84)
- Numerical Pruning for Efficient Autoregressive Models (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.