Research map / Learning theory and algorithms / Combinatorial and search optimization
On Warm-Starting Neural Network Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| warm-start from half-data model instead of random init | ResNet-18 Adam · CIFAR-10 val | accuracy (%): 78.0 -> 74.4 | Hurt |
| warm-start instead of random init | Logistic regression · CIFAR-10 val | accuracy (%): 33.8 -> 33.3 | No clear effect |
| warm-start instead of random init | MLP Adam · SVHN val | accuracy (%): 76.7 -> 69.4 | Hurt |
| add L2 regularization 1e-1 in both rounds (ablation) | warm-started ResNet-18 · CIFAR-10 val | accuracy (%): RI 72.7 vs WS 63.9 | No clear effect |
| add adversarial training and confidence penalty (ablation) | warm-started ResNet-18 · CIFAR-10 val | accuracy (%): RI 54.8 vs WS 52.4 (adv 1e-1) | No clear effect |
| warm start last layer first, then full training (ablation) | warm-started ResNet-18 · CIFAR-10 val | accuracy (%): 52.5 -> 50.9 | Hurt |
| add noise only (no shrinking) to warm-started params (ablation) | warm-started ResNet-18 · CIFAR-10 val | accuracy (%): 50.8 -> 54.4; train time 13.6 -> 165.3 min | Mixed |
| tune lr and batch size to close the gap (ablation) | warm-started ResNet-18 · CIFAR-10 val | accuracy/time | No clear effect |
| apply shrink and perturb initialization (lambda ~0.6) instead of pure warm start | ResNet-18 · CIFAR-10 online learning | validation accuracy | Helped |
| add perturbation noise after shrinking (ablation) | shrink-perturb ResNet-18 · CIFAR-10 online learning | accuracy and train time | Helped |
| shrink only last-layer weights instead of all (ablation) | ResNet-18 · CIFAR-10 online learning | validation accuracy | Hurt |
| apply shrink-perturb instead of warm start or random init in pre-training transfer | ResNet-18 · pre-training to varying target fraction | accuracy | Helped |
Closest papers by meaning
- Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training (NeurIPS 2024 · similarity 0.88)
- Where Do Large Learning Rates Lead Us? (NeurIPS 2024 · similarity 0.87)
- Why Warmup the Learning Rate? Underlying Mechanisms and Improvements (NeurIPS 2024 · similarity 0.87)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.87)
- DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticity (NeurIPS 2024 · similarity 0.86)
- Algorithm Development in Neural Networks: Insights from the Streaming Parity Task (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.