Research map / Optimization / Nonconvex and smooth optimization
Variance Reduction for Faster Non-Convex Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SVRG variance reduction instead of full gradient descent | Smooth finite-sum non-convex optimization · Smooth finite-sum non-convex optimization | stochastic gradient iterations: Ω(n1/3) times faster than GD | Helped |
| Use SVRG for non-convex sigmoid-loss ERM | ℓ2-regularized ERM · ERM with sigmoid loss | testing accuracy | Mixed |
| Use SVRG rather than SGD for sigmoid-loss ERM | ℓ2-regularized ERM · ERM with sigmoid loss | training objective | Mixed |
| Use SVRG-based algorithms rather than SGD or AdaGrad | Toy neural networks · CIFAR-10 and MNIST | training objective | Helped |
| Initialize each epoch with a weighted average of recent iterates (ablation) | SVRG-1 · CIFAR-10 and MNIST | training objective | Helped |
| Use adaptive learning rates (ablation) | SVRG-2 · CIFAR-10 and MNIST | training objective | Helped |
| Reduce mini-batch size to 16 (ablation) | SVRG-3 · CIFAR-10 and MNIST | training objective | Helped |
| Reduce mini-batch size (ablation) | SGD or AdaGrad · CIFAR-10 and MNIST | training objective | Hurt |
Closest papers by meaning
- Stochastic Variance Reduction for Nonconvex Optimization (ICML 2016 · similarity 0.92)
- Momentum-Based Variance Reduction in Non-Convex SGD (NeurIPS 2019 · similarity 0.92)
- Stochastic Optimization with Variance Reduction for Infinite Datasets with Finite Sum Structure (NeurIPS 2017 · similarity 0.91)
- Simple Stochastic Gradient Methods for Non-Smooth Non-Convex Regularized Optimization (ICML 2019 · similarity 0.91)
- Double Variance Reduction: A Smoothing Trick for Composite Optimization Problems without First-Order Gradient (ICML 2024 · similarity 0.89)
- Random Scaling and Momentum for Non-smooth Non-convex Optimization (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.