Research map / Architectures and efficiency / Pruning and efficient tuning
Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that dynamic sparse training (SET, RigL, MEST, GraNet) at low sparsity (10–50%) outperforms dense training in corruption robustness across image and video benchmarks, and attributes this to implicit regularization that reduces sensitivity to high-frequency content.
Problem
Whether dense training is the best choice for maximizing robustness to image corruptions, and whether dynamic sparse training, which is usually studied for efficiency, can improve corruption robustness at no extra resource cost.
Approach
Proposes the Dynamic Sparsity Corruption Robustness (DSCR) hypothesis: DST at low sparsity levels yields more corruption-robust models than dense training, explained by implicit regularization from the remove-regrow dynamics that de-emphasizes high-frequency features.
Key results
- SET at sparsity 0.5 improves robustness on 18 of 19 CIFAR100-C corruptions relative to dense ResNet34.
- In the gradient-regrow summary table, every DST method (RigL, MEST_g, GraNet_g) beats dense training across all 9 scenarios, e.g., ImageNet-C ResNet50 38.38 dense vs 38.72 GraNet_g at sparsity 0.1.
- On ImageNet-R, RigL reaches 40.04% vs 37.84% for dense ResNet50.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with RigL at sparsity 0.4 instead of dense training | VGG16 · CIFAR10-C | robustness accuracy (%): 73.32 -> 73.64 | Helped |
| Train with RigL at sparsity 0.4 instead of dense training | ResNet34 · CIFAR100-C | robustness accuracy (%): 51.65 -> 52.56 | Helped |
| Train with RigL at sparsity 0.4 instead of dense training | EfficientNet-B0 · TinyImageNet-C | robustness accuracy (%): 15.42 -> 16.46 | Helped |
| Train with RigL at sparsity 0.1 instead of dense training | ResNet50 · ImageNet-C | robustness accuracy (%): 38.38 -> 38.70 | Helped |
| Train with RigL at sparsity 0.1 instead of dense training | ResNet50 · ImageNet-C-bar | robustness accuracy (%): 40.38 -> 41.44 | Helped |
| Train with RigL at sparsity 0.1 instead of dense training | ResNet50 · ImageNet-3DCC | robustness accuracy (%): 43.62 -> 44.08 | Helped |
| Train with MEST_g at sparsity 0.1 instead of dense training | DeiT-base · ImageNet-C | robustness accuracy (%): 54.68 -> 55.15 | Helped |
| Train with GraNet_g at sparsity 0.1 instead of dense training | 3D ResNet50 · UCF101 | robustness accuracy (%): 51.14 -> 53.57 | Helped |
| Train with MEST_g at sparsity 0.1 instead of dense training | I3D · UCF101 | robustness accuracy (%): 53.58 -> 56.3 | Helped |
| Use SET instead of dense training | MLP · Fashion-MNIST-C | robustness accuracy (%): 19.82 ± 0.32 -> 23.44 ± 1.79 | Helped |
| Use RigL instead of dense training | ResNet50 · ImageNet-R | average classification accuracy (%): 37.84 -> 40.04 | Helped |
| Add Mixup to SET training | CIFAR100 model · CIFAR100 | robustness accuracy (%): 52.60 -> 54.93 | Helped |
Surprising findings
- Sparse models at moderate sparsity beat dense models on corruption robustness while also reducing training and memory cost.
- Dense models with the highest resource usage do not necessarily yield better robustness.
Limitations
- Efficiency gains are theoretical; binary masks simulate sparsity and current hardware does not fully exploit it.
- Mainly studies vanilla DST at low sparsity; high-sparsity regime not explored.
- Mechanistic explanation (frequency-based implicit regularization) is largely empirical and suggestive.
Experimental setup
Datasets: CIFAR10 CIFAR100 TinyImageNet ImageNet UCF101 Fashion-MNIST Pascal VOC 2012
Benchmarks: CIFAR10-C CIFAR100-C TinyImageNet-C ImageNet-C ImageNet-Cbar ImageNet-3DCC UCF101-C Fashion-MNIST-C ImageNet-R ImageNet-v2
Metrics: robustness accuracy relative robustness gain training FLOPs inference FLOPs parameter count
Baselines: Dense training AC/DC Cannistraci-Hebb training
Builds on: Sparse Evolutionary Training RigL MEST GraNet AugMix
Closest papers by meaning
- SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training (ICML 2026 · similarity 0.88)
- Navigating Extremes: Dynamic Sparsity in Large Output Spaces (NeurIPS 2024 · similarity 0.87)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.86)
- Adaptive Sharpness-Aware Pruning for Robust Sparse Networks (ICLR 2024 · similarity 0.86)
- Sparse Cocktail: Every Sparse Pattern Every Sparse Ratio All At Once (ICML 2024 · similarity 0.86)
- Advancing Dynamic Sparse Training by Exploring Optimization Opportunities (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.