Research map / Optimization / Nonconvex and smooth optimization
New Insight into Hybrid Stochastic Gradient Descent: Beyond With-Replacement Sampling and Convexity
Pan Zhou, Xiaotong Yuan, Jiashi Feng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use without-replacement sampling instead of with-replacement sampling in HSGD (ablation) | HSGD · letter, softmax regression, single pass | objective value | Improved |
| HSGD with exponentially growing mini-batch vs SGD, SVRG, SAGA, AVRG, SCGC | HSGD (WoRS) · Logistic and softmax regression, single epoch, 8 datasets | objective distance log(f - f*) | Improved |
| HSGD vs variance-reduced baselines on ill-conditioned softmax regression (lambda=0.001), multi-epoch | HSGD (WoRS) · protein, satimage, sensorless, letter | objective distance log(f - f*) | Improved |
| Use HSGD-exp (exponential batch growth) for training a 3-layer neural network | HSGD · ijcnn1, covtype, sensorless feedforward NN | training loss | Improved |
| Linearly instead of exponentially increasing mini-batch size in HSGD (ablation) | HSGD on neural networks · ijcnn1, covtype | solution accuracy | Mixed |
Closest papers by meaning
- On the Convergence of mSGD and AdaGrad for Stochastic Optimization (ICLR 2022 · similarity 0.89)
- A Hybrid Stochastic-Deterministic Minibatch Proximal Gradient Method for Efficient Optimization and Generalization (TPAMI 2021 · similarity 0.89)
- Incremental Gradient Descent with Small Epoch Counts is Surprisingly Slow on Ill-Conditioned Problems (ICML 2025 · similarity 0.89)
- Closing the convergence gap of SGD without replacement (ICML 2020 · similarity 0.89)
- Demystifying SGD with Doubly Stochastic Gradients (ICML 2024 · similarity 0.87)
- The Sample Complexity of Gradient Descent in Stochastic Convex Optimization (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
