Research map / Continual learning / Online and replay-based
DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DASH, a direction-aware weight-shrinking initialization that forgets memorized noise while keeping learned features, to fix warm-start loss of plasticity under stationary data.
Problem
Warm-starting neural networks from previously learned weights when new i.i.d. data arrives in a stationary distribution hurts generalization (loss of plasticity), and the mechanism behind this is poorly understood.
Approach
An abstract discrete feature-learning framework attributes warm-start degradation to noise memorization; DASH shrinks each neuron's incoming weight vector by a factor based on cosine similarity with the negative gradient on cumulative data, so weights that learned features are kept while noise-driven weights are forgotten.
Key results
- On Tiny-ImageNet with ResNet-18 (SGD), DASH reaches 46.11% final test accuracy vs 25.69% for random init and 34.34% for S&P, with fewer steps (8341 vs 30237 for random init).
- On CIFAR-10 with ResNet-18 (SGD), DASH reaches 84.08% final test accuracy vs 67.32% for random init and 81.25% for S&P.
- On SVHN with ResNet-18 (SGD), DASH reaches 93.67% vs 86.27% for random init; in the SoTA setting DASH is close to but does not exceed cold-start.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DASH instead of S&P | ResNet-18 training with SGD · Tiny-ImageNet | test accuracy (last experiment): 34.34 (0.48) -> 46.11 (0.34) | Helped |
| Apply DASH instead of S&P | ResNet-18 training with SGD · CIFAR-10 | test accuracy (last experiment): 81.25 (0.14) -> 84.08 (0.52) | Helped |
| Apply DASH instead of S&P | ResNet-18 training with SGD · CIFAR-100 | test accuracy (last experiment): 50.08 (0.23) -> 57.99 (0.28) | Helped |
| Apply DASH instead of S&P | ResNet-18 training with SGD · SVHN | test accuracy (last experiment): 92.67 (0.17) -> 93.67 (0.13) | Helped |
| Apply DASH instead of random initialization | ResNet-18 training with SGD · Tiny-ImageNet | test accuracy (last experiment): 25.69 (0.13) -> 46.11 (0.34) | Helped |
| Apply DASH instead of random initialization | ResNet-18 training with SGD · CIFAR-10 | test accuracy (last experiment): 67.32 (0.51) -> 84.08 (0.52) | Helped |
| Apply DASH instead of random initialization | ResNet-18 training with SGD · CIFAR-100 | test accuracy (last experiment): 35.52 (0.14) -> 57.99 (0.28) | Helped |
| Apply DASH instead of random initialization | ResNet-18 training with SGD · SVHN | test accuracy (last experiment): 86.27 (0.46) -> 93.67 (0.13) | Helped |
| Apply DASH instead of S&P | ResNet-18 training with SGD · CIFAR-10 | number of steps (last experiment): 5395 (625) -> 6490 (399) | Hurt |
| Apply DASH instead of S&P | ResNet-18 training with SGD · SVHN | number of steps (last experiment): 3597 (156) -> 5161 (672) | Hurt |
| Apply DASH instead of random initialization | ResNet-18 training with SGD · Tiny-ImageNet | number of steps (last experiment): 30237 (368) -> 8341 (368) | Helped |
| Apply DASH instead of random initialization | ResNet-18 training with SGD · CIFAR-100 | number of steps (last experiment): 10557 (247) -> 3519 (0) | Helped |
Surprising findings
- Techniques for non-stationary plasticity loss (L2 init, Reset, layer normalization, dead-neuron revival) fail to help in the stationary warm-start setting.
- Warm-starting with SAM does not outperform cold-starting with SAM.
Limitations
- In the SoTA training setting DASH approaches but does not outperform cold-starting.
- Theory relies on an abstract discrete feature-learning framework rather than optimization dynamics of real networks.
- Applying DASH at every data arrival can hurt efficiency in continual data streams; an interval is needed.
Experimental setup
Datasets: Tiny-ImageNet CIFAR-10 CIFAR-100 SVHN ImageNet-1k
Benchmarks: Tiny-ImageNet CIFAR-10 CIFAR-100 SVHN ImageNet-1k
Metrics: test accuracy number of update steps to 99.9% train accuracy
Baselines: Random Init (cold-start) Warm Init Shrink & Perturb L2 INIT Reset Warm ReM
Builds on: Shrink & Perturb feature learning theory (Allen-Zhu & Li) sharpness-aware minimization
Closest papers by meaning
- Local Redundancy: An Information-Theoretic Measure of Plasticity from Synthetic Memorization (ICML 2026 · similarity 0.87)
- On Warm-Starting Neural Network Training (NeurIPS 2020 · similarity 0.86)
- Self-Normalized Resets for Plasticity in Continual Learning (ICLR 2025 · similarity 0.86)
- The Pitfalls of Memorization: When Memorization Hurts Generalization (ICLR 2025 · similarity 0.85)
- Forget Sharpness: Perturbed Forgetting of Model Biases Within SAM Dynamics (ICML 2024 · similarity 0.85)
- PaLoRA: Paced Low-Rank Adaptation for Continual Learning (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.