Research map / Architectures and efficiency / Pruning and efficient tuning
UniPTS: A Unified Framework for Proficient Post-Training Sparsity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniPTS, a post-training sparsity framework combining a base-decayed KL sparsity objective, evolutionary reducing-regrowing sparsity distribution search, and iteration-wise dynamic sparse training, achieving large gains over POT at high sparsity on ImageNet and VOC.
Problem
Post-training sparsity (PTS) with only a small calibration set suffers severe accuracy collapse at high sparsity ratios compared with full-data retraining methods.
Approach
Transposes three factors of conventional sparsity into PTS: a global KL objective whose log base decays over training, an evolutionary search for layerwise sparsity with reducing-regrowing sparsification and noisy-calibration fitness to avoid overfitting, and per-iteration dynamic sparse training with magnitude-based masks and decayed pruned weights.
Key results
- On ResNet-50 at 90% sparsity on ImageNet-1K, UniPTS reaches 68.60% top-1 versus 3.90% for POT.
- On Faster-RCNN and SSD at 90% sparsity on PASCAL VOC, UniPTS improves mAP over POT by 3.3 and 3.4 points.
- UniPTS also improves N:M sparsity (e.g., ResNet-50 2:8 pattern: 71.46% vs 47.24% for POT).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UniPTS instead of POT for ResNet-50 pruning at 90% sparsity | ResNet-50 · ImageNet-1K | Top-1 accuracy (%): 3.90 -> 68.60 | Helped |
| Use UniPTS instead of POT for SSD pruning at 90% sparsity | SSD · PASCAL VOC | mAP: 57.02 -> 64.42 | Helped |
| Use UniPTS instead of POT for Faster-RCNN pruning at 90% sparsity | Faster-RCNN · PASCAL VOC | mAP: 51.29 -> 54.59 | Helped |
| Use a base-decayed KL sparsity objective instead of global MSE, cross-entropy, normal KL, or dynamic temperature (ablation) | UniPTS · ImageNet-1K, ResNet-50 | Top-1 accuracy (%): 13.07; 38.99; 64.47; 64.54 -> 68.64 | Helped |
| Use searched sparsity distribution instead of ERK or learnable distribution (ablation) | UniPTS · ImageNet-1K, ResNet-50, 90% sparsity | Top-1 accuracy (%): 66.97; 64.48 -> 68.60 | Helped |
| Use searched sparsity distribution instead of L2-norm distribution (ablation) | POT · ImageNet-1K, ResNet-50, 90% sparsity | Top-1 accuracy (%): 3.88 -> 5.07 | Helped |
| Use iteration-wise mask updates instead of longer update intervals (ablation) | UniPTS · ImageNet-1K, ResNet-50, 90% sparsity | Top-1 accuracy (%): 66.86% to 57.51% when α equals 0 and the interval increases from 100 to 1000 | Helped |
| Increase the pruned-weight decay factor α from zero to a suitable nonzero value (ablation) | UniPTS · ImageNet-1K, ResNet-50, 90% sparsity | Top-1 accuracy (%) | Mixed |
Surprising findings
- Iteration-wise mask updates (interval 1) perform best, contrary to conventional DST practice where longer intervals help, because longer intervals overfit the small calibration set.
- Learnable sparsity distribution (STR) collapses to nearly uniform sparsity and performs poorly in the post-training setting.
Limitations
- Evaluated only on computer vision tasks (classification and detection) with CNNs.
- Pruning search is considerably slower than heuristic allocation (ERK variant), trading accuracy for time.
- Hyperparameters such as decay rate gamma and alpha were set and tuned on the evaluation setting.
Experimental setup
Datasets: ImageNet-1K PASCAL VOC
Benchmarks: ImageNet-1K PASCAL VOC
Metrics: Top-1 accuracy mAP pruning time
Baselines: POT RigL STR ERK
Builds on: POT RigL STR EagleEye
Closest papers by meaning
- Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in Minutes (AAAI 2024 · similarity 0.83)
- ELSA: Exploiting Layer-wise N:M Sparsity for Vision Transformer Acceleration (NeurIPS 2024 · similarity 0.82)
- Distributed Sparse Regression via Penalization (NeurIPS 2024 · similarity 0.80)
- Navigating Extremes: Dynamic Sparsity in Large Output Spaces (NeurIPS 2024 · similarity 0.79)
- DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs (NeurIPS 2025 · similarity 0.79)
- Sparse Cocktail: Every Sparse Pattern Every Sparse Ratio All At Once (ICML 2024 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.