Research map / Architectures and efficiency / Pruning and efficient tuning
Dynamic Sparse Training of Diagonally Sparse Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DynaDiag, a differentiable TopK-based dynamic sparse training method that learns diagonal sparsity patterns, with a BCSR-based GPU kernel for real training and inference speedups.
Problem
Unstructured sparsity in dynamic sparse training matches dense accuracy but yields little wall-clock speedup on GPUs, while existing structured sparse training methods lose accuracy at high sparsity and often use dense backpropagation.
Approach
Parameterize each weight matrix as a sum of K permutation-placed diagonals whose positions are chosen by a temperature-annealed differentiable TopK over learnable importance scores with an l1 penalty; diagonals are converted to block CSR so both forward and backward passes are sparse and hardware-friendly.
Key results
- On ViT-L/16 at 90% sparsity on ImageNet-1K, DynaDiag reaches 77.74% top-1, about 2.28 points above the next best structured method (SRigL).
- At 90% sparsity on ViT-Base, DynaDiag achieves up to 3.13x inference speedup and 1.59x training speedup over the dense equivalent.
- On WikiText-103 at 90% sparsity, GPT2-Small with DynaDiag is 6.22 perplexity points better than the next best DST method, within 2.57 of RigL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add BCSR conversion of diagonal matrices for the GPU kernel (vs direct diagonal computation) (ablation) | DynaDiag ViT-B/16 at 90% sparsity · ImageNet-1K | training time (hrs): 18.07 -> 11.42 hrs | Helped |
| add BCSR conversion of diagonal matrices (vs direct diagonal computation) (ablation) | DynaDiag ViT-B/16 at 90% sparsity · ImageNet-1K | Top-1 accuracy: 76.92 vs 76.91 | No clear effect |
| diagonal structured sparsity via TopK-based DST instead of unstructured RigL | ViT-L/16 at 90% · ImageNet-1K | Top-1 accuracy: 77.74 vs 78.26 (RigL) | Hurt |
| diagonal structured sparsity vs SRigL (N:M structured) at 90% | ViT-L/16 · ImageNet-1K | Top-1 accuracy: 77.74 vs 75.46 | Helped |
| diagonal structured sparsity vs DSB/PixelatedBFly/DiagHeur at 90% on ViT-B/16 | ViT-B/16 at 90% · ImageNet-1K | Top-1 accuracy: 76.91 vs 75.90 (SRigL), 73.89 (PixelatedBFly), 72.89 (DSB), 71.46 (DiagHeur) | Helped |
| diagonal structured sparsity at 95% on ViT-B/16 and ViT-L/16 | DynaDiag at 95% sparsity · ImageNet-1K | Top-1 accuracy: 69.54 vs RigL 71.50 (ViT-B/16); 69.59 vs 72.11 (ViT-L/16) | Hurt |
| diagonal structured sparsity vs unstructured RigL on GPT2-Small at 90% | GPT2-S at 90% sparsity · WikiText-103 | perplexity: 56.33 vs 53.76 (RigL), but 10.1 lower than PixelatedBFly 66.44 | Hurt |
| diagonal structured sparsity at 90% sparsity inference and training on GPU vs dense | ViT-B/16 at 90% · GPU wall-clock | inference speedup: up to 3.13x inference, 1.59x training | Helped |
| diagonal structured sparsity at 60% sparsity on GPU | ViT-B/16 at 60% · GPU wall-clock | training speedup: 0.98x training at 60% | No clear effect |
| LoRA-FA fine-tuning (rank 6) added on top of diagonally sparse ViT-B/16 at 80% (ablation) | DynaDiag ViT-B/16 at 80% · ImageNet-1K | Top-1 accuracy: surpasses RigL at rank 6 with 1.67% more parameters | Helped |
Surprising findings
- DynaDiag surpasses RigL at extreme sparsities (up to 99.99%) on ImageNet-1K.
- Adding LoRA-FA low-rank fine-tuning (about 1.67% extra parameters) to a diagonally sparse ViT-B/16 at 80% sparsity exceeds RigL accuracy.
Limitations
- Scales poorly to CNNs due to the cost of searching distinct diagonal patterns per channel.
- Training speedup is near parity or slower at lower sparsity (0.98x at 60%), and the PyTorch implementation does not use optimized CUDA kernels.
- Accuracy still trails RigL at sparsities of 80% and above on several models without the LoRA-FA fine-tuning.
Experimental setup
Datasets: ImageNet-1K CIFAR-10 CIFAR-100 WikiText-103
Benchmarks: ImageNet-1K CIFAR-10 CIFAR-100 WikiText-103
Metrics: Top-1 accuracy perplexity inference wall-clock speedup training wall-clock speedup McNemar test p-value
Baselines: RigL SET CHT CHTs MEST SRigL PixelatedBFly DSB DiagHeur
Builds on: RigL Top-KAST SRigL Pixelated Butterfly SmaT
Closest papers by meaning
- Dynamic Sparse Training with Structured Sparsity (ICLR 2024 · similarity 0.92)
- Navigating Extremes: Dynamic Sparsity in Large Output Spaces (NeurIPS 2024 · similarity 0.89)
- Advancing Dynamic Sparse Training by Exploring Optimization Opportunities (ICML 2024 · similarity 0.88)
- Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency (ICML 2024 · similarity 0.86)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.86)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.