Research map / Optimization / Optimizers
SEBOOST - Boosting Stochastic Learning Using Subspace Optimization Techniques
Elad Richardson, Rom Herskovitz, Boris Ginsburg, Michael Zibulevsky
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SEBOOST subspace optimization on top of SGD | SGD baseline · MNIST autoencoder | reconstruction error / convergence | Improved |
| add SEBOOST subspace optimization on top of NAG | NAG baseline · MNIST autoencoder | reconstruction error / convergence | Improved |
| add SEBOOST subspace optimization on top of AdaGrad | AdaGrad baseline · MNIST autoencoder | reconstruction error / convergence | Improved |
| add SEBOOST subspace optimization on top of SGD and NAG | SGD and NAG baselines · CIFAR-10 ResNet-32 | final minimum / accuracy | Improved |
| add SEBOOST subspace optimization on top of AdaGrad | AdaGrad baseline · CIFAR-10 ResNet-32 | final minimum / accuracy | No clear effect |
| add SEBOOST on top of SGD with a simple regression network | SGD baseline · Simple regression (20,000 samples, 6->12->8->4->1) | convergence speed | Improved |
| apply subspace optimization too frequently (small ell) (ablation) | SEBOOST · MNIST autoencoder | runtime and final minimum | Hurt |
| increase M from small to 50 (larger subspace) (ablation) | SEBOOST · MNIST autoencoder | final minimum vs early descent speed | Mixed |
| add anchor points to the subspace (ablation) | SEBOOST · MNIST autoencoder | reconstruction error / convergence | Improved |
| add momentum direction to the subspace (ablation) | SEBOOST · MNIST autoencoder | convergence | Mixed |
| change baseline learning rate (robustness test) (ablation) | SEBOOST boosted variants · MNIST autoencoder | convergence sensitivity to learning rate | Improved |
| apply boosting step once per epoch vs too frequently or too infrequently (ablation) | SEBOOST on ResNet-32 · CIFAR-10 ResNet-32 | final minimum | Mixed |
Closest papers by meaning
- SnapBoost: A Heterogeneous Boosting Machine (NeurIPS 2020 · similarity 0.86)
- NGBoost: Natural Gradient Boosting for Probabilistic Prediction (ICML 2020 · similarity 0.86)
- A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance (ICML 2021 · similarity 0.85)
- Minimal Variance Sampling in Stochastic Gradient Boosting (NeurIPS 2019 · similarity 0.85)
- Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (NeurIPS 2019 · similarity 0.85)
- Improving Neural Network Training in Low Dimensional Random Bases (NeurIPS 2020 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
