Research map / Optimization / Nonconvex and smooth optimization
Stochastic Variance Reduction for Nonconvex Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SVRG with fixed step size and epoch structure instead of SGD on nonconvex finite-sum problems | nonconvex finite-sum problems · IFO complexity to epsilon-accurate stationary point | IFO complexity: O(n + n^{2/3}/eps) vs O(1/eps^2) for SGD | Helped |
| Use SVRG instead of full gradient descent on nonconvex finite-sum problems | nonconvex finite-sum problems · IFO complexity to epsilon-accurate stationary point | IFO complexity: n^{1/3} factor faster than GradientDescent | Helped |
| Use gradient-dominated SVRG variant (GD-SVRG) for global linear convergence | tau-gradient dominated functions · IFO complexity to epsilon accuracy | IFO complexity: O((n + tau n^{2/3}) log(1/eps)) vs O(n + n tau log(1/eps)) for GD | Helped |
| Use mini-batch of size b in SVRG (b < n^{2/3}) with step size scaled by b (ablation) | nonconvex SVRG · theoretical IFO complexity / parallel speedup | convergence factor: rate improves by factor b; IFO complexity unchanged | Helped |
| Use SGD mini-batching instead of SVRG mini-batching | SGD · mini-batch convergence | convergence rate: O(1/sqrt(bT)) dependence, worse than SVRG's 1/b | Hurt |
| Use SVRG with n-dependent step size eta=1/(8L sqrt(n)) in convex case versus n-independent step 1/(8L) (ablation) | convex SVRG · IFO complexity | IFO complexity: O(n + sqrt(n)/eps) vs O(n/eps) | Helped |
| Use SVRG on neural network training (one hidden layer) with fixed step, mini-batch 10, SGD warm start | one-hidden-layer neural networks on CIFAR-10, MNIST, STL-10 · gradient norm and training loss | ||grad f||^2 and training loss | Helped |
| Use SVRG instead of SGD for CIFAR-10 test error | one-hidden-layer neural network on CIFAR-10 · test error | test error | Helped |
| Use SVRG instead of SGD for MNIST and STL-10 test error | one-hidden-layer neural network on MNIST and STL-10 · test error | test error | No clear effect |
| Use SVRG (n-independent) for strongly convex f with nonconvex f_i, rather than SDCA | l2-regularized ERM with nonconvex f_i · IFO complexity | IFO complexity: O((n + n^{2/3} kappa) log(1/eps)) vs SDCA O((n + kappa^2) log(1/eps)) when kappa > n^{2/3} | Helped |
Closest papers by meaning
- Stochastic Nested Variance Reduction for Nonconvex Optimization (NeurIPS 2018 · similarity 0.94)
- Proximal Stochastic Methods for Nonsmooth Nonconvex Finite-Sum Optimization (NeurIPS 2016 · similarity 0.92)
- Variance Reduction for Faster Non-Convex Optimization (ICML 2016 · similarity 0.92)
- Hybrid Stochastic-Deterministic Minibatch Proximal Gradient: Less-Than-Single-Pass Optimization with Nearly Optimal Generalization (ICML 2020 · similarity 0.92)
- Parallel Asynchronous Stochastic Variance Reduction for Nonconvex Optimization (AAAI 2017 · similarity 0.91)
- High-probability complexity bounds for stochastic non-convex minimax optimization (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.