Research map / Architectures and efficiency / Pruning and efficient tuning
Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in Minutes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FCPTS, a post-training sparsity framework that learns per-layer sparsity rates via a KDE-based differentiable bridge from threshold to sparsity, with a control loss enforcing a global sparsity target, reconstructing the network in minutes.
Problem
Post-training sparsity (PTS) suffers large accuracy loss at high sparsity because existing sparsity-rate allocation methods are designed for retraining and fail to converge stably with limited calibration data.
Approach
Builds a differentiable bridge from magnitude threshold to layer sparsity rate using kernel density estimation of weight distributions, and adds a control loss that drives the weighted global sparsity to a specified target, enabling net-wise joint optimization of weights and per-layer sparsity in one pass.
Key results
- On ImageNet ResNet-50 at 80% sparsity, FCPTS reaches 71.26% top-1 vs. 22.92% for POT (ERK), an improvement of over 30 points.
- At 70% sparsity on ResNet-18, FCPTS achieves 68.50% top-1 vs. 64.28% for POT (ERK) and 46.99% for ProbMask.
- ImageNet ResNet-18 sparse model generation takes 29 minutes vs. 100 minutes for POT on the stated hardware.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FCPTS instead of POT (L2Norm) | ResNet-32 · CIFAR-10, 70% sparsity | Top1 accuracy: 91.06 -> 92.91 | Helped |
| Use FCPTS instead of POT (ERK) | ResNet-32 · CIFAR-100, 90% sparsity | Top1 accuracy: 4.66 -> 63.35 | Helped |
| Use FCPTS instead of POT (ERK) | ResNet-18 · ImageNet, 80% sparsity | Top1 accuracy: 50.78 -> 65.83 | Helped |
| Use FCPTS instead of POT (ERK) | ResNet-50 · ImageNet, 80% sparsity | Top1 accuracy: 22.92 -> 71.26 | Helped |
| Use FCPTS instead of POT(ERK) | ViT base · ImageNet, 70% sparsity | Top1 accuracy: 30.50 -> 65.24 | Helped |
| Use FCPTS instead of POT (ERK) | MobileNetV2 SSD-Lite · PASCAL VOC, 90% sparsity | mAP: 0.3 -> 59.1 | Helped |
| Use FCPTS instead of POT | ResNet-18 · ImageNet reconstruction | time (min): 100 -> 29 | Helped |
| Increase sparsity from dense to 70% | ResNet-18 · CV22 inference | speedup: - -> 2.394 | Helped |
| Increase sparsity from dense to 70% | ResNet-18 · CV22 inference | latency (ms): 18.194 -> 7.599 | Helped |
| Increase sparsity from dense to 70% | ResNet-18 · CV22 inference | memory (MB): 11.632 -> 5.245 | Helped |
| Learn layer sparsity rates instead of keeping the ERK-initialized rates fixed (ablation) | ResNet-50, RegNetX-400M, and MobileNetV2 · Multiple sparsity rates; especially RegNetX-400M at 80% sparsity | accuracy: about 10% improvement | Helped |
Surprising findings
- STR fails to produce valid results under the post-training setting, and ProbMask underperforms the naive post-training baseline POT.
- Learnable sparsity allocation is reported to rescue fixed ERK allocations, with about 10% gain for RegNetX-400M at 80% sparsity.
Limitations
- Relies on a small calibration set (10k images), and results depend on this limited data regime.
- Reported accuracy drops are still notable at very high sparsity for some architectures (e.g., MobileNetV2 at 90% on ImageNet, 13.99% top-1).
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet PASCAL VOC
Benchmarks: CIFAR-10 CIFAR-100 ImageNet PASCAL VOC
Metrics: Top-1 accuracy Top-5 accuracy mAP reconstruction time (min) latency (ms) memory (MB)
Baselines: POT (L2Norm) POT (ERK) ProbMask STR
Builds on: POT STR ProbMask ERK
Closest papers by meaning
- Spartan: Differentiable Sparsity via Regularized Transportation (NeurIPS 2022 · similarity 0.89)
- Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together! (ICLR 2023 · similarity 0.89)
- Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (NeurIPS 2022 · similarity 0.89)
- Sparse Cocktail: Every Sparse Pattern Every Sparse Ratio All At Once (ICML 2024 · similarity 0.87)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.86)
- Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.