Research map / Architectures and efficiency / Pruning and efficient tuning
Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DST instead of dense training | LeNet-300-100 · MNIST | accuracy: 98.16 ± 0.06 -> 97.69±0.14; -0.47 | Hurt |
| Use DST instead of dense training | LeNet-5-Caffe · MNIST | accuracy: 99.18 ± 0.05 -> 99.11±0.07; -0.07 | Hurt |
| Use DST instead of dense training | LSTM-a · MNIST | accuracy: 98.64 ± 0.12 -> 98.70±0.06; +0.06 | Helped |
| Use DST instead of dense training | LSTM-b · MNIST | accuracy: 98.87 ± 0.07 -> 98.89±0.11; +0.02 | Helped |
| Use DST rather than Sparse Momentum | VGG-16 · CIFAR-10 | accuracy: 93.36 ± 0.04 -> 93.93 ± 0.05 | Helped |
| Use DST rather than Sparse Momentum | WideResNet-16-8 · CIFAR-10 | accuracy: 94.87 ± 0.04 -> 95.05 ± 0.08 | Helped |
| Use DST rather than DSR | WideResNet-16-8 · CIFAR-10 | accuracy: 94.93 ± 0.04 -> 95.05 ± 0.08 | Helped |
| Use DST rather than Sparse Momentum | ResNet-50 · ImageNet (ILSVRC2012) | top-1 accuracy: 73.80 / 91.80 -> 74.02 / 92.49 | Helped |
| Increase sparse-regularization coefficient α (ablation) | DST · CIFAR-10 | model remaining ratio | Hurt |
| Use a very large α that reduces remaining percentage below 5% (ablation) | DST · CIFAR-10 | accuracy | Hurt |
| Use a trainable threshold vector rather than a scalar threshold (ablation) | DST trainable masked layers · Various architectures | robustness | Helped |
| Use a trainable threshold vector rather than a threshold matrix (ablation) | DST trainable masked layers · Various architectures | additional parameter storage: less than 1% additional network parameter | Helped |
Closest papers by meaning
- Training Your Sparse Neural Network Better with Any Mask (ICML 2022 · similarity 0.92)
- The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training (ICLR 2022 · similarity 0.92)
- Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace Offset (ICML 2021 · similarity 0.91)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.90)
- Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization (ICML 2019 · similarity 0.89)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.