Research map / Label-efficient and robust learning / Self-supervised and contrastive
The Early Phase of Neural Network Training
Jonathan Frankle, David J. Schwab, Ari S. Morcos
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Rewind IMP sub-networks to iteration 500 or later | ResNet-20 · CIFAR-10 | accuracy at 16.8% of weights remaining: 16.8% | Improved |
| Use iteration-500 signs with initialization magnitudes instead of iteration-500 signs and magnitudes (ablation) | IMP-derived ResNet-20 sub-network · CIFAR-10 | accuracy | Hurt |
| Use iteration-500 magnitudes with initialization signs (ablation) | IMP-derived ResNet-20 sub-network · CIFAR-10 | accuracy | Improved |
| Use iteration-2000 signs with initialization magnitudes (ablation) | IMP-derived ResNet-20 sub-network · CIFAR-10 | accuracy | Mixed |
| Shuffle iteration-k weights globally or within layers (ablation) | IMP-derived ResNet-20 sub-network · CIFAR-10 | accuracy | Hurt |
| Shuffle weights within filters rather than globally or layerwise (ablation) | IMP-derived ResNet-20 sub-network · CIFAR-10 | accuracy: 0.0 ± 0.092; 0.0±0.143; 0.0±0.144 | Mixed |
| Restrict shuffling to weights with the same sign (ablation) | IMP-derived ResNet-20 sub-network · CIFAR-10 | accuracy: 0.0±0.049 | Improved |
| Add Gaussian noise to iteration-500 weights (ablation) | IMP-derived ResNet-20 sub-network · CIFAR-10 | accuracy: 0.5σ to 1σ | Hurt |
| Add Gaussian noise to iteration-2000 weights (ablation) | IMP-derived ResNet-20 sub-network · CIFAR-10 | accuracy: 1σ; 2σ | Mixed |
| Pre-train with random labels (ablation) | ResNet-20 · CIFAR-10 | IMP sub-network accuracy: up to 10 epochs | Hurt |
| Pre-train with self-supervised rotation prediction instead of supervised target training (ablation) | ResNet-20 · CIFAR-10 | IMP sub-network accuracy: 40 compared to 1.25; 32× | Improved |
| Pre-train on 4x-blurred inputs with correct labels (ablation) | ResNet-20 · CIFAR-10 | IMP sub-network accuracy: 40 epochs; 500 iterations (1.25 epochs) | Improved |
Closest papers by meaning
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.89)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.87)
- Understanding and Leveraging the Learning Phases of Neural Networks (AAAI 2024 · similarity 0.87)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.87)
- On the Weight Dynamics of Deep Normalized Networks (ICML 2024 · similarity 0.87)
- The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
