AI Paper Atlas

Research map / Optimization / Implicit bias and generalization

Predicting Training Time Without Training

Luca Zancato, Alessandro Achille, Avinash Ravichandran, Rahul Bhotika, Stefano Soatto

NeurIPS 2020 · Optimization › Implicit bias and generalization · approach: Edge-of-stability dynamics · 10 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use a linearized network SDE in function space to predict fine-tuning timePre-trained ResNet fine-tuning · Multiple datasets and hyperparametersTT prediction error: within a 20\% error marginImproved
Use the function-space predictor rather than training the networkPre-trained ResNet fine-tuning · Experiments across datasetsruntime: 30-45-fold reduction in costImproved
Use the function-space predictor for GD training-time estimationResNet-18 fine-tuned from ImageNet pre-training · Several target datasets and hyperparametersTT prediction accuracy: within 13\% of the real training time 95\% of the timesImproved
Use the function-space predictor for SGD training-time estimationResNet-18 fine-tuned from ImageNet pre-training · Several target datasets and hyperparametersTT prediction accuracy: within 20\% of the real training timeImproved
Use a larger learning rateLinearized training-time predictor · Fine-tuningprediction accuracyHurt
Use a smaller batch size, increasing the stochastic componentLinearized training-time predictor · SGD fine-tuningprediction errorHurt
Use different learning-rate and momentum pairs with the same effective learning rateFine-tuned network · Fine-tuning loss curvesloss-curve similarityNo clear effect
Use gradients rather than features to represent and cluster samplesPre-trained network · CIFAR-10 subset with 3 classesclass separationImproved
Predict a larger dataset's training curve from a smaller sample subsetLinearized training-time predictor · MSE loss, CIFAR-10loss-curve predictionImproved
Remove batch normalization and prevent its statistics from being updated (ablation)Linearized predictor · GD and SGD fine-tuninglinearization approximation qualityImproved
Remove data augmentation (ablation)Linearized predictor · GD and SGD fine-tuninglinearization approximation qualityImproved
Keep the estimated SGD noise constant instead of rescaling it with the loss (ablation)SDE predictor · CIFAR-10 error-curve predictionasymptotic behavior predictionHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy