Research map / Architectures and efficiency / RNNs and normalization
Advancing Dynamic Sparse Training by Exploring Optimization Opportunities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BiDST, which reformulates dynamic sparse training as a bi-level optimization problem and learns the sparse mask via gradient-based (implicit gradient, Gumbel-Softmax) updates alongside SGD weight training, reducing mask-update overhead.
Problem
Dynamic sparse training (DST) updates sparse masks with heuristic prune-and-grow rules, which yields suboptimal mask search efficiency, early mask stagnation, and high system overhead from frequent topology changes.
Approach
Views the train-prune-grow schedule of DST as a bi-level problem and optimizes the mask directly through the network loss using implicit gradients (first-order approximation) and Gumbel-Softmax relaxation, rather than heuristic mask updates; the mask is binarized only periodically to cut overhead.
Key results
- On CIFAR-10 with ResNet-32 at 98% sparsity, BiDST reaches 91.99%/92.21% accuracy, up to 2.62% above the best baseline (e.g., RigL-ITOP 90.40%).
- On ImageNet-1K ResNet-50 at 90% uniform sparsity, BiDST reaches 75.3% top-1 with 18 mask updates versus RigL's 72.0% with 320 updates at similar training FLOPs.
- On a Snapdragon 888 device, BiDST achieves 1.88x-2.14x training speedup over dense training and 16-25x reduced mask-update overhead relative to DST baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BiDST's jointly optimized mask and weights instead of heuristic dynamic sparse training | ResNet-32 · CIFAR-10, ERK sparsity | accuracy: 93.70 -> 94.12±0.11; 92.78 -> 93.26±0.12; 90.40 -> 92.21±0.09 | Helped |
| Use BiDST's jointly optimized mask and weights instead of heuristic dynamic sparse training | ResNet-32 · CIFAR-100, ERK sparsity | accuracy: 71.16 -> 73.42±0.14; 69.38 -> 71.50±0.07; 66.35 -> 68.19 ±0.15 | Helped |
| Use BiDST instead of the strongest reported baseline | ResNet-50 · ImageNet-1K, 80% and 90% sparsity | Top-1 accuracy: 76.7 -> 76.1; 75.9 -> 75.3 | Mixed |
| Use BiDST sparse training instead of dense training | ResNet-32, VGG-19, and ResNet-50 on-device training · Snapdragon 888 | speedup: CPU: 2.01×, GPU: 2.14×; CPU: 1.88×, GPU: 2.04×; CPU: 1.88×, GPU: 1.83× | Helped |
| Use BiDST's reduced mask-update frequency instead of frequent DST mask updates | Dynamic sparse training on Snapdragon 888 · On-device training overhead | overhead reduction: 16-25× reduced overhead time | Helped |
Surprising findings
- Larger mask search space and larger regularization coefficient are not necessarily better; the authors report that the search space size is not monotonic in benefit.
Limitations
- Mask search space S and regularization coefficient λ require hyperparameter tuning, and the best search space is not monotonic.
- Experiments limited to CIFAR and ImageNet image classification with CNN architectures.
- The first-order approximation (dropping second-order weight Hessian term) is a heuristic simplification of the exact implicit gradient.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1K
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1K
Metrics: top-1 accuracy training FLOPs inference FLOPs number of mask updates training speedup system overhead intersection over union of masks
Baselines: SNIP GraSP DeepR SET DSR RigL RigL-ITOP MEST SNFS Top-KAST C-GaP SpFDE
Builds on: RigL MEST SET GraSP Gumbel-Softmax
Closest papers by meaning
- Navigating Extremes: Dynamic Sparsity in Large Output Spaces (NeurIPS 2024 · similarity 0.90)
- Dynamic Sparse Training with Structured Sparsity (ICLR 2024 · similarity 0.89)
- Dynamic Sparse Training of Diagonally Sparse Networks (ICML 2025 · similarity 0.88)
- Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency (ICML 2024 · similarity 0.88)
- Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness (ICLR 2025 · similarity 0.86)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.