Research map / Optimization / Optimizers
Scalable Adaptive Stochastic Optimization Using Random Projections
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use random-projection full-matrix preconditioning instead of diagonal preconditioning | RADAGRAD vs. ADAGRAD · Penn Treebank | training loss: 1.52 × 10−3 -> 5.62 × 10−4 | Helped |
| Use random-projection full-matrix preconditioning instead of diagonal preconditioning | RADAGRAD vs. ADAGRAD · Penn Treebank | test loss: 3.23 × 10−2 -> 1.15 × 10−2 | Helped |
| Add SVRG variance reduction (ablation) | RADAGRAD · MNIST, CIFAR-10, and SVHN | training loss | Helped |
| Add SVRG variance reduction (ablation) | RADAGRAD · MNIST, CIFAR-10, and SVHN | test accuracy | No clear effect |
| Use random-projection full-matrix preconditioning instead of diagonal preconditioning | RADAGRAD vs. ADAGRAD · MNIST, CIFAR-10, and SVHN | training loss | Helped |
| Use random-projection full-matrix preconditioning instead of diagonal preconditioning | RADAGRAD vs. ADAGRAD · simulated low-effective-rank classification data | training loss | Helped |
| Add the corrected gradient update for the component outside the projected subspace (ablation) | RADAGRAD · deep-network optimization | optimization performance | Helped |
| Add exponential forgetting schemes similar to RMSPROP and ADAM (ablation) | RADAGRAD · optimization experiments | performance | Hurt |
Closest papers by meaning
- SGD with AdaGrad Stepsizes: Full Adaptivity with High Probability to Unknown Parameters, Unbounded Gradients and Affine Variance (ICML 2023 · similarity 0.89)
- AdaGrad under Anisotropic Smoothness (ICLR 2025 · similarity 0.89)
- Universality of AdaGrad Stepsizes for Stochastic Optimization: Inexact Oracle, Acceleration and Variance Reduction (NeurIPS 2024 · similarity 0.87)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.87)
- Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad (NeurIPS 2024 · similarity 0.87)
- Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guarantees (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.