Research map / Optimization / Optimizers
Fast Variance Reduction Method with Stochastic Batch Size
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SAGA++'s mixture of full-gradient and single-sample updates instead of SAGA or SVRG | Variance-reduced solvers for ℓ1-regularized logistic regression · kddb, avazu, and criteo | running time | Helped |
| Use SAGA++ instead of SAGA or SVRG | ℓ1-regularized logistic regression · kddb, varying regularization parameters | running time | Helped |
| Choose batch size 1 at every step instead of a larger average batch (ablation) | Stochastic-batch variance reduction framework · Strongly convex finite-sum optimization | convergence rate per data access | Helped |
| Account for cache and disk I/O by mixing full-gradient and single-sample updates | SAGA++ · ℓ1-regularized logistic regression on kddb, avazu, and criteo | running time | Helped |
| Keep control variates updated during single-sample steps rather than leaving snapshots stale between full-gradient steps (ablation) | SAGA++ compared with SVRG-style updates · Variance-reduced optimization | convergence rate | Helped |
| Apply lazy updates for ℓ1 regularization (ablation) | SAGA, SVRG, and SAGA++ · Sparse-data optimization | running time | Helped |
| Use SAGA rather than SVRG (ablation) | Variance-reduced solvers · Experiments on the paper's datasets | running time | Hurt |
| Use only stochastic updates rather than including sequential full-gradient access (ablation) | Variance-reduced solvers · Experiments on the paper's datasets | running time | Hurt |
| Avoid lazy updates (ablation) | Variance-reduced methods for ℓ1 regularization · Sparse-data optimization | running time | Hurt |
Closest papers by meaning
- Optimal Mini-Batch and Step Sizes for SAGA (ICML 2019 · similarity 0.90)
- Lightweight Stochastic Optimization for Minimizing Finite Sums with Infinite Data (ICML 2018 · similarity 0.87)
- An Improved Analysis and Rates for Variance Reduction under Without-replacement Sampling Orders (NeurIPS 2021 · similarity 0.85)
- Nonconvex Variance Reduced Optimization with Arbitrary Sampling (ICML 2019 · similarity 0.84)
- Adaptive Variance Reduction for Stochastic Optimization under Weaker Assumptions (NeurIPS 2024 · similarity 0.82)
- Subsampling is not Magic: Why Large Batch Sizes Work for Differentially Private Stochastic Optimisation (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.