Research map / Architectures and efficiency / Pruning and efficient tuning
Dynamic Sparse Training with Structured Sparsity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SRigL, a sparse-to-sparse dynamic sparse training method that learns constant fan-in (fine-grained structured) sparsity with a neuron ablation rule, and shows real CPU/GPU inference speedups from the condensed representation.
Problem
Dynamic sparse training yields unstructured sparsity that is hard to accelerate on real hardware, while structured sparse pruning often hurts generalization at high sparsity.
Approach
Enforce a constant fan-in constraint inside RigL-style dynamic sparse training so every neuron has the same number of nonzero weights, and add a neuron ablation rule (threshold gamma_sal on salient weights per neuron) that lets high-sparsity layers drop neurons, matching RigL's generalization while enabling a condensed representation for real speedups.
Key results
- On ResNet-50/ImageNet at 90% sparsity (1x training), SRigL with ablation reaches 72.7% top-1 vs RigL 72.8%; at 80% it reaches 75.0% vs 74.9%.
- On ResNet-18/CIFAR-10 at 99% sparsity, SRigL with ablation achieves 92.8±0.1 vs RigL 92.9±0.1 (w/o ablation 91.5±0.3).
- A 90% sparse linear layer gives real-world speedups of 3.4x on CPU for online inference and 1.7x on GPU at batch size 256 over dense layers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add constant fan-in sparsity with neuron ablation instead of unstructured RigL | ResNet-18 · CIFAR-10, 99% sparsity | test accuracy: $92.9\pm 0.1$ -> $92.8\pm 0.1$ | No clear effect |
| Add constant fan-in sparsity with neuron ablation instead of unstructured RigL | ResNet-50 · ImageNet, 90% sparsity, 1× training | top-1 test accuracy: $72.8$ -> $72.7$ | No clear effect |
| Add constant fan-in sparsity with neuron ablation instead of unstructured RigL | ViT-B/16 · ImageNet, 90% sparsity | top-1 test accuracy: $76.4$ -> $76.0$ | No clear effect |
| Add neuron ablation to constant-fan-in SRigL (ablation) | ResNet-18 · CIFAR-10, 99% sparsity | test accuracy: $91.5\pm 0.3$ -> $92.8\pm 0.1$ | Helped |
| Add neuron ablation to constant-fan-in SRigL (ablation) | ResNet-50 · ImageNet, 99% sparsity, 1× training | top-1 test accuracy: $48.7$ -> $51.5$ | Helped |
| Add neuron ablation to constant-fan-in SRigL (ablation) | ViT-B/16 · ImageNet, 90% sparsity | top-1 test accuracy: $71.3$ -> $76.0$ | Helped |
| Vary the neuron-ablation threshold γ_sal (ablation) | SRigL on ResNet-18 and Wide ResNet-22 · CIFAR-10, high sparsities | generalization performance | No clear effect |
| Use a high neuron-ablation threshold γ_sal of 90% to 99% (ablation) | SRigL on ViT · ImageNet | generalization performance | Helped |
| Use constant fan-in sparsity with the condensed representation | SRigL linear layer · CPU online inference, 90% sparse layer | speedup: 3.4× compared to dense; 2.5× compared to CSR | Helped |
| Use constant fan-in sparsity with the condensed representation | SRigL linear layer · GPU inference, batch size 256, 90% sparse layer | speedup: 1.7× compared to dense; 13.0× compared to CSR | Helped |
Surprising findings
- RigL at high sparsity (>90%) ablates whole neurons, effectively reducing layer width, which motivated the neuron ablation mechanism.
- Constant fan-in sparsity produces slightly smaller output-norm variance than constant-per-layer or Bernoulli sparsity, contrary to the expectation that structure would hurt training.
Limitations
- Speedups come from a straightforward PyTorch/CUDA implementation rather than optimized kernels or hardware support.
- Gamma_sal hyperparameter needed tuning; ViT required a different value (95%) and was sensitive to it.
- Single seed for ResNet-50 ImageNet results; dense first layer used in one RigL comparison point.
Experimental setup
Datasets: CIFAR-10 ImageNet
Benchmarks: CIFAR-10 ImageNet
Metrics: top-1 accuracy wall-clock inference time speedup over dense and CSR FLOPs
Baselines: RigL SET DeepR DSR Top-KAST MEST SNFS DSB Chase SR-STE AST Static sparse
Builds on: RigL SR-STE ERK sparsity distribution Chase
Closest papers by meaning
- Dynamic Sparse Training of Diagonally Sparse Networks (ICML 2025 · similarity 0.92)
- Navigating Extremes: Dynamic Sparsity in Large Output Spaces (NeurIPS 2024 · similarity 0.90)
- Advancing Dynamic Sparse Training by Exploring Optimization Opportunities (ICML 2024 · similarity 0.89)
- Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency (ICML 2024 · similarity 0.88)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.87)
- A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.