Research map / Optimization / Nonconvex and smooth optimization
A Simple Stochastic Variance Reduced Algorithm with Fast Convergence Rates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MiG's Katyusha-style averaging momentum without Nesterov momentum | SVRG · Strongly convex problems with m/κ ≤ 3/4 | convergence rate | Helped |
| Add Nesterov momentum to MiG (ablation) | MiG · Strongly convex problems with m/κ ≤ 3/4 | oracle calls | Mixed |
| Add Nesterov momentum using Acc-Prox-SVRG (ablation) | SVRG · Strongly convex problems with m/κ ≤ 3/4 | convergence | Helped |
| Add Nesterov momentum using Acc-Prox-SVRG (ablation) | SVRG · Strongly convex problems with m/κ > 3/4 | convergence | Hurt |
| Use Katyusha with its suggested parameter restriction (ablation) | Katyusha · Strongly convex problems with m/κ > 3/4 | convergence | Hurt |
| Tune both τ2 and τ1 in Katyusha (ablation) | Katyusha · Strongly convex problems with m/κ > 3/4 | convergence | Helped |
| Use the last inner-loop iterate rather than the snapshot as the next epoch's starting vector (ablation) | MiG · Strongly convex problems | oracle complexity: \bigO{(n+\kappa)\log{({1}/{\epsilon})}} | Helped |
| Replace weighted averaging with uniform averaging (ablation) | Sparse and asynchronous MiG · Sparse and asynchronous problems | practical performance | Helped |
| Use Option II fake averaging instead of Option I averaging (ablation) | Asynchronous Sparse MiG · Sparse asynchronous experiments | oracle calls | Helped |
| Use MiG instead of KroMagnon or ASAGA | Sparse asynchronous optimization · RCV1 and KDD2010 | oracle calls | Helped |
| Use MiG instead of Katyusha | Strongly convex problems with relatively large λ · Serial dense experiments | oracle calls and running time | Helped |
Closest papers by meaning
- Stochastic Optimization with Variance Reduction for Infinite Datasets with Finite Sum Structure (NeurIPS 2017 · similarity 0.91)
- Dissipativity Theory for Accelerating Stochastic Variance Reduction: A Unified Analysis of SVRG and Katyusha Using Semidefinite Programs (ICML 2018 · similarity 0.90)
- Variance Reduction for Faster Non-Convex Optimization (ICML 2016 · similarity 0.90)
- Stochastic Variance Reduction for Nonconvex Optimization (ICML 2016 · similarity 0.90)
- Variance Reduction via Accelerated Dual Averaging for Finite-Sum Optimization (NeurIPS 2020 · similarity 0.89)
- Double Variance Reduction: A Smoothing Trick for Composite Optimization Problems without First-Order Gradient (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.