Research map / Architectures and efficiency / RNNs and normalization
DSD²: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes distilling a sparse (or dense) teacher into a pruned student to avoid sparse double descent, and uses a neuron-state entropy measure to re-enable early stopping.
Problem
Iterative magnitude pruning of over-parameterized networks shows a sparse double descent (SDD) curve, so test accuracy first worsens, then improves, then collapses, which breaks traditional early-stopping criteria for selecting the sparsity level.
Approach
Train the pruned student with knowledge distillation (KL + CE loss) from a sparse teacher at its best validation region (or a dense teacher), so the teacher's response acts as an implicit regularizer that makes the student's accuracy vs. sparsity curve monotonic; additionally track the average neuron activation entropy on the training set, which stays flat then decreases in the classical regime, t
Key results
- On CIFAR-10 with 20% label noise, the distilled-from-sparse-teacher student reaches 86.89 ± 0.16% test accuracy at 99.26% sparsity, versus 74.52 ± 1.20% for the vanilla dense-trained-then-pruned baseline at 99.62% sparsity, with 35.82 vs 48.84 PFLOPs training cost.
- On CIFAR-10 with 50% label noise, distillation from a pruned teacher reaches 75.87 ± 0.18% at 99.80% sparsity, versus 64.37 ± 1.89% for the vanilla baseline.
- On CIFAR-10 with 20% noise, the best validation model shifts from the light phase at low noise (below 15%) to the sweet phase at higher noise.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill the student from a dense teacher instead of training it with vanilla early stopping | VGG-like student · CIFAR-10, ε=20% | test accuracy: 60.23 ± 3.37 -> 81.52 ± 1.85 | Helped |
| distill from a pruned teacher instead of a dense teacher | VGG-like student · CIFAR-10, ε=20% | test accuracy: 81.52 ± 1.85 -> 86.89 ± 0.16 | Helped |
| use knowledge distillation during iterative pruning | VGG-like student · CIFAR-10 and CIFAR-100, ε ∈ {10%, 20%, 50%} | sparse double descent | Helped |
| distill from a sparse, well-performing teacher rather than a dense teacher | VGG-like student · CIFAR-10, ε=20% | test accuracy: 81.52 ± 1.85 -> 86.89 ± 0.16 | Helped |
| randomly re-initialize pruned weights instead of leaving them unperturbed (ablation) | ResNet-18 · CIFAR-100, ε=10% | test accuracy | Hurt |
| rewind pruned weights to their original initialization instead of leaving them unperturbed (ablation) | ResNet-18 · CIFAR-100, ε=10% | test accuracy | Mixed |
| scale down the learning rate while leaving learned parameters unperturbed (ablation) | pruned model · CIFAR-100, ε=10% | test accuracy | Hurt |
| increase model depth (ablation) | VGG-like model · CIFAR-10 | sparse double descent | Hurt |
| increase model width (ablation) | VGG-like model · CIFAR-10 | sparse double descent | Hurt |
| use knowledge distillation rather than vanilla training as width or depth increases (ablation) | VGG-like student · CIFAR-10, ε=50% | sparse double descent | Helped |
| use data augmentation, dropout, or the tested ℓ1/ℓ2 regularization (ablation) | ResNet-18 · CIFAR-100, ε=50% | sparse double descent | No clear effect |
| increase ℓ1 regularization from λ=1e-5 to λ=1e-4 (ablation) | ResNet-18 · CIFAR-100, ε=50% | training success | Hurt |
Surprising findings
- Distilling from a non-pruned (dense) teacher also avoids sparse double descent for the student.
- Standard l1, l2, dropout and data augmentation regularization did not remove SDD in the reported experiments.
Limitations
- Requires training a large teacher model, which may be unaffordable in resource-constrained settings.
- Experiments are limited to image classification with small CNNs and ResNets on CIFAR-scale and ImageNet data.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 CIFAR-100N Flowers-102 ImageNet
Benchmarks: CIFAR-10 CIFAR-100 CIFAR-100N Flowers-102 ImageNet
Metrics: test accuracy sparsity training FLOPs (PFLOPs)
Baselines: vanilla pruned student (no distillation) early stopping on vanilla student l1 regularization l2 regularization dropout data augmentation (AutoAugment, CutOut)
Builds on: lottery ticket hypothesis knowledge distillation (Hinton et al.) iterative magnitude pruning information bottleneck theory
Closest papers by meaning
- Sparse Double Descent: Where Network Pruning Aggravates Overfitting (ICML 2022 · similarity 0.91)
- On the Role of Optimization in Double Descent: A Least Squares Study (NeurIPS 2021 · similarity 0.89)
- Unraveling the Enigma of Double Descent: An In-depth Analysis through the Lens of Learned Feature Space (ICLR 2024 · similarity 0.88)
- On Double Descent in Reinforcement Learning with LSTD and Random Features (ICLR 2024 · similarity 0.85)
- Asymmetric Scaling Laws from Sparse Features (NeurIPS 2026 · similarity 0.84)
- Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorization (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.