Research map / Optimization / Implicit bias and generalization
Predicting Training Time Without Training
Luca Zancato, Alessandro Achille, Avinash Ravichandran, Rahul Bhotika, Stefano Soatto
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a linearized network SDE in function space to predict fine-tuning time | Pre-trained ResNet fine-tuning · Multiple datasets and hyperparameters | TT prediction error: within a 20\% error margin | Improved |
| Use the function-space predictor rather than training the network | Pre-trained ResNet fine-tuning · Experiments across datasets | runtime: 30-45-fold reduction in cost | Improved |
| Use the function-space predictor for GD training-time estimation | ResNet-18 fine-tuned from ImageNet pre-training · Several target datasets and hyperparameters | TT prediction accuracy: within 13\% of the real training time 95\% of the times | Improved |
| Use the function-space predictor for SGD training-time estimation | ResNet-18 fine-tuned from ImageNet pre-training · Several target datasets and hyperparameters | TT prediction accuracy: within 20\% of the real training time | Improved |
| Use a larger learning rate | Linearized training-time predictor · Fine-tuning | prediction accuracy | Hurt |
| Use a smaller batch size, increasing the stochastic component | Linearized training-time predictor · SGD fine-tuning | prediction error | Hurt |
| Use different learning-rate and momentum pairs with the same effective learning rate | Fine-tuned network · Fine-tuning loss curves | loss-curve similarity | No clear effect |
| Use gradients rather than features to represent and cluster samples | Pre-trained network · CIFAR-10 subset with 3 classes | class separation | Improved |
| Predict a larger dataset's training curve from a smaller sample subset | Linearized training-time predictor · MSE loss, CIFAR-10 | loss-curve prediction | Improved |
| Remove batch normalization and prevent its statistics from being updated (ablation) | Linearized predictor · GD and SGD fine-tuning | linearization approximation quality | Improved |
| Remove data augmentation (ablation) | Linearized predictor · GD and SGD fine-tuning | linearization approximation quality | Improved |
| Keep the estimated SGD noise constant instead of rescaling it with the loss (ablation) | SDE predictor · CIFAR-10 error-curve prediction | asymptotic behavior prediction | Hurt |
Closest papers by meaning
- Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks (ICLR 2018 · similarity 0.87)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.87)
- A Convergence Theory for Deep Learning via Over-Parameterization (ICML 2019 · similarity 0.86)
- SGD with Large Step Sizes Learns Sparse Features (ICML 2023 · similarity 0.86)
- Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup (NeurIPS 2019 · similarity 0.86)
- Chaotic Dynamics are Intrinsic to Neural Network Training with SGD (NeurIPS 2022 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
