Research map / Architectures and efficiency / RNNs and normalization
Dynamic Neural Regeneration: Enhancing Deep Learning Generalization on Small Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dynamic Neural Regeneration (DNR), an iterative training method that uses SNIP-based data-aware dynamic masking each generation to choose which parameters to reinitialize, improving generalization on small datasets.
Problem
Training deep networks on small datasets leads to overfitting; existing iterative weight-reinitialization methods (e.g., Knowledge Evolution) use random, fixed masks that may be suboptimal.
Approach
Replace KE's fixed random fit/reset mask with a mask recomputed at the start of each generation using SNIP connection-sensitivity on a data subset, reinitializing a fixed 20% of parameters with the lowest saliency.
Key results
- On small datasets with ResNet18 and label smoothing, DNR (f10) reaches a mean of 67.45% across five datasets versus 63.17% for Smth + KE (f10) and 65.89% for Smth + LB (f10).
- On CUB-200 with ResNet18, DNR (f10) achieves 71.37% vs 66.51% for KE (f10).
- On large datasets with ResNet50, Smth + DNR (f10) reaches mean 74.72% vs 74.10% for Smth (f1) and 73.33% for Smth + KE (f10).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace a fixed random mask with data-aware dynamic SNIP masking | DNR versus KE · small datasets (CUB-200-2011, FGVC-Aircraft, Stanford Dogs, Flower102, MIT67) | accuracy: DNR (f10): 59.72 ±0.21 vs KE (f10): 58.11 ±0.25 (CUB); 55.87 ±0.47 vs 53.21 ±0.43 (Aircraft); 65.76±0.13 vs 64.56±0.31 ( | Helped |
| use DNR instead of the equal-computation long baseline | ResNet50 · CIFAR10 | accuracy: 93.60±0.29 -> 94.61±0.30 | Helped |
| use DNR instead of the equal-computation long baseline | ResNet50 · CIFAR100 | accuracy: 74.21±0.28 -> 75.05±0.23 | Helped |
| use DNR instead of the equal-computation long baseline | ResNet50 · Tiny-ImageNet | accuracy: 51.16 ±0.21 -> 54.50 ±0.26 | Helped |
| use DNR instead of vanilla transfer learning | ResNet18 · CUB-200-2011 | accuracy: 65.63 ±0.21 -> 68.56 ±0.24 | Helped |
| use SNIP instead of random selection (ablation) | DNR · CUB-200-2011 | accuracy: 66.51±0.07 -> 71.87 ±0.22 | Helped |
| use SNIP instead of Fisher Importance (ablation) | DNR · CUB-200-2011 | accuracy: 67.73 ±0.28 -> 71.87 ±0.22 | Helped |
| use SNIP instead of weight magnitude (ablation) | DNR · CUB-200-2011 | accuracy: 64.18 ±0.19 -> 71.87 ±0.22 | Helped |
| vary the reinitialized parameter fraction from 5% to 40% (ablation) | DNR · FGVC-Aircraft and CUB-200-2011 | accuracy: Aircraft: 65.34, 66.10, 66.63, 64.13, 62.79; CUB: 69.95, 70.15, 71.37, 68.42, 66.87 | Mixed |
| reduce importance-estimation data from 0.2 |D| to 128 samples (ablation) | DNR · FGVC-Aircraft and CUB-200-2011 | accuracy: Aircraft: 66.63 -> 66.45; CUB: 71.37 -> 71.26 | No clear effect |
Surprising findings
- Weight-magnitude importance performed worse than random selection for choosing parameters to reinitialize (64.18% vs 66.51% on CUB200).
- KE underperformed the long baseline trained for equal computation on small datasets.
Limitations
- Performance is sensitive to the choice of parameter selection/importance criterion, as the authors note.
- Gains on large datasets (CIFAR, Tiny-ImageNet) are comparable or small, and the medical experiment is preliminary with limited hyperparameter tuning.
- Reinitialization ratio is fixed at 20% globally, and gains saturate with more generations (Flower f30 vs f20).
Experimental setup
Datasets: CUB-200-2011 Flower-102 MIT67 Stanford Dogs FGVC-Aircraft CIFAR-10 CIFAR-100 Tiny-ImageNet CIFAR-10-C PAPILA
Benchmarks: CUB-200-2011 Flower-102 MIT67 Stanford Dogs FGVC-Aircraft CIFAR-10 CIFAR-100 Tiny-ImageNet CIFAR-10-C PGD-10 adversarial attack
Metrics: accuracy mean corruption accuracy adversarial accuracy balanced accuracy sensitivity specificity
Baselines: Cross-entropy (long baseline) DSD BAN Knowledge Evolution (KE) LW LLF Label smoothing Transfer learning (vanilla fine-tuning)
Builds on: Knowledge Evolution SNIP Born Again Networks Dense-Sparse-Dense Forget and relearn (LLF)
Closest papers by meaning
- NICE: Neurogenesis Inspired Contextual Encoding for Replay-free Class Incremental Learning (CVPR 2024 · similarity 0.86)
- Optimal Rates for Generalization of Gradient Descent for Deep ReLU Classification (NeurIPS 2025 · similarity 0.84)
- Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization (NeurIPS 2024 · similarity 0.84)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.84)
- Task-Free Continual Generation and Representation Learning via Dynamic Expansionable Memory Cluster (AAAI 2024 · similarity 0.84)
- RG-GAN: Dynamic Regenerative Pruning for Data-Efficient Generative Adversarial Networks (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.