AI Paper Atlas

Research map / Label-efficient and robust learning / Self-supervised and contrastive

The Early Phase of Neural Network Training

Jonathan Frankle, David J. Schwab, Ari S. Morcos

ICLR 2020 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Invariance-based self-supervised learning · 47 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Rewind IMP sub-networks to iteration 500 or laterResNet-20 · CIFAR-10accuracy at 16.8% of weights remaining: 16.8%Improved
Use iteration-500 signs with initialization magnitudes instead of iteration-500 signs and magnitudes (ablation)IMP-derived ResNet-20 sub-network · CIFAR-10accuracyHurt
Use iteration-500 magnitudes with initialization signs (ablation)IMP-derived ResNet-20 sub-network · CIFAR-10accuracyImproved
Use iteration-2000 signs with initialization magnitudes (ablation)IMP-derived ResNet-20 sub-network · CIFAR-10accuracyMixed
Shuffle iteration-k weights globally or within layers (ablation)IMP-derived ResNet-20 sub-network · CIFAR-10accuracyHurt
Shuffle weights within filters rather than globally or layerwise (ablation)IMP-derived ResNet-20 sub-network · CIFAR-10accuracy: 0.0 ± 0.092; 0.0±0.143; 0.0±0.144Mixed
Restrict shuffling to weights with the same sign (ablation)IMP-derived ResNet-20 sub-network · CIFAR-10accuracy: 0.0±0.049Improved
Add Gaussian noise to iteration-500 weights (ablation)IMP-derived ResNet-20 sub-network · CIFAR-10accuracy: 0.5σ to 1σHurt
Add Gaussian noise to iteration-2000 weights (ablation)IMP-derived ResNet-20 sub-network · CIFAR-10accuracy: 1σ; 2σMixed
Pre-train with random labels (ablation)ResNet-20 · CIFAR-10IMP sub-network accuracy: up to 10 epochsHurt
Pre-train with self-supervised rotation prediction instead of supervised target training (ablation)ResNet-20 · CIFAR-10IMP sub-network accuracy: 40 compared to 1.25; 32×Improved
Pre-train on 4x-blurred inputs with correct labels (ablation)ResNet-20 · CIFAR-10IMP sub-network accuracy: 40 epochs; 500 iterations (1.25 epochs)Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy