Research map / Learning theory and algorithms / Kernels and regression theory
How many degrees of freedom do we need to train deep networks: a loss landscape perspective
Brett W Larsen, Stanislav Fort, Nic Becker, Surya Ganguli
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the training dimension of a random affine subspace | Random affine subspace training · MNIST, Fashion MNIST, CIFAR-10, CIFAR-100, and SVHN | success probability of reaching a training loss sublevel set | Improved |
| Require a lower target loss | Random and burn-in affine subspace training · MNIST, Fashion MNIST, CIFAR-10, and CIFAR-100 | threshold training dimension | Hurt |
| Increase burn-in steps before restricted training (ablation) | Burn-in affine subspace training · MNIST, Fashion MNIST, CIFAR-10, and CIFAR-100 | threshold training dimension | Improved |
| Use burn-in initialization for Conv-3 on MNIST (ablation) | Conv-3 affine subspace training · MNIST | accuracy | No clear effect |
| Train in a lottery subspace formed from principal components of the full training trajectory | Lottery subspace training · MNIST, Fashion MNIST, CIFAR-10, CIFAR-100, and SVHN | accuracy at low training dimensions | Improved |
| Use lottery subspaces rather than random affine subspaces | Subspace training · MNIST, Fashion MNIST, CIFAR-10, CIFAR-100, and SVHN | training dimension needed to attain full accuracy: 20 to 60 | Improved |
| Use lottery tickets rather than random affine subspaces | Conv-2 on CIFAR-10 · CIFAR-10 | accuracy | Improved |
| Use lottery tickets rather than random affine subspaces | Lottery ticket pruning · Experiments where methods overlap | accuracy | No clear effect |
| Increase lottery-ticket compression to high ratios | Lottery ticket pruning · High-compression settings | accuracy | Hurt |
| Replace the full network with its NTK linearization around a local optimum (ablation) | Conv-2 · MNIST and Fashion MNIST | threshold training dimension | No clear effect |
Closest papers by meaning
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.89)
- Large Scale Structure of Neural Network Loss Landscapes (NeurIPS 2019 · similarity 0.89)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.87)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.87)
- Implicit bias of SGD in $L_2$-regularized linear DNNs: One-way jumps from high to low rank (ICLR 2024 · similarity 0.87)
- How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspective (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
