Research map / Optimization / Implicit bias and generalization
Learning Curves for SGD on Structured Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Gaussian-feature theory to predict test-loss dynamics | Random-feature models and wide neural networks · MNIST and CIFAR-10 | test loss | Helped |
| Use structured rather than isotropic features | ReLU random-feature models · MNIST and CIFAR-10 | test loss | Helped |
| Increase neural-network depth | Fully connected neural networks · MNIST | learning speed | Helped |
| Use a CNN rather than a standard MLP | Neural networks · CIFAR-10 | power-law exponent | Helped |
| Choose a small minibatch at fixed compute when the learning rate is optimized | SGD on structured features · Fixed compute budget | test loss | Helped |
| Choose an intermediate minibatch size rather than the smallest batch | SGD on realistic feature spectra · Fixed compute budget | test loss | Mixed |
| Increase minibatch size | SGD on structured features · Fixed number of steps | test loss | Helped |
| Replace the mode-coupling term with a decoupled diagonal approximation (ablation) | SGD theory for power-law features · Fixed compute budget | batch-size prediction accuracy | Hurt |
| Approximate empirical feature fourth moments with Gaussian fourth moments | ReLU random-feature models · Random-feature model | fourth-moment matrix discrepancy: ~ 5% | No clear effect |
| Train on a finite training set and evaluate on a separate test distribution | Random-feature models · MNIST | training error | Helped |
| Train on a finite training set rather than fresh samples from the full distribution | Random-feature models · MNIST test set | test error | Hurt |
| Increase learning rate | SGD on ReLU random features · Realistic data | test loss | Mixed |
Closest papers by meaning
- On the Double Descent of Random Features Models Trained with SGD (NeurIPS 2022 · similarity 0.89)
- On the interplay between data structure and loss function in classification problems (NeurIPS 2021 · similarity 0.89)
- SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochs (NeurIPS 2021 · similarity 0.88)
- Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning (ICML 2024 · similarity 0.88)
- Learning Curves of Stochastic Gradient Descent in Kernel Regression (ICML 2025 · similarity 0.87)
- Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Features (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.