Research map / Optimization / Nonconvex and smooth optimization
Stacey: Promoting Stochastic Steepest Descent via Accelerated $\ell_p$-Smooth Nonconvex Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Stacey, an accelerated l_p steepest descent optimizer that interpolates primal (l_p steepest descent) and dual (mirror/gradient) iterates, with O(eps^-4) stochastic non-convex guarantees, and evaluates it against SGD, Adam, AdamW, and Lion on CIFAR/ImageNet image classification and LLaMA-100M C4 pretraining.
Problem
Standard optimizers (SGD, AdamW, Lion) rely on l2 or l_infinity steepest descent geometry, leaving general l_p norms with p in (2, infinity) underexplored for stochastic non-convex training and acceleration.
Approach
Accelerate stochastic l_p steepest descent by interpolating two iterate sequences (a primal l_p steepest descent step and a dual mirror/gradient step) via a linear-coupling-style convex combination, with variants pairing the dual step with an l2 or l_p distance-generating function.
Key results
- On CIFAR at 200 epochs, Stacey(p,p) reaches 93.79% test accuracy vs 92.76% for SGD with momentum and 92.66% for Adam.
- On LLaMA-100m C4 pretraining at 30K steps, Stacey(p,p) reaches test loss 4.4879 vs 5.4240 for AdamW and 6.7733 for Lion.
- On ImageNet at 60 epochs, Stacey(p,p) gets 69.87% top-1 vs 69.31% for AdamW and 69.62% for Lion, while Stacey(p,2) lags at 67.56%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Stacey instead of Adam, AdamW, or Lion | ResNet-18 · CIFAR test @200 epoch | Testing ACC (%): 92.66\std{0.36} -> 93.79\std{0.38} | Helped |
| use Stacey instead of Lion or AdamW | ResNet-50 · ImageNet Top-1 test @60 epoch | Testing Top-1 ACC (%): 69.62\std{0.11} -> 69.87\std{0.14} | Helped |
| use Stacey instead of AdamW | llama-100m · C4 pretraining test @30k steps | Testing Loss: 5.4240\std{0.0093} -> 4.4879\std{0.0132} | Helped |
| use the (p,p) variant instead of (p,2) (ablation) | Stacey on ResNet-18 · CIFAR test @200 epoch | Testing ACC (%): 93.54\std{0.06} -> 93.79\std{0.38} | Helped |
| use the (p,p) variant instead of (p,2) (ablation) | Stacey on ResNet-50 · ImageNet Top-1 test @60 epoch | Testing Top-1 ACC (%): 67.56\std{0.15} -> 69.87\std{0.14} | Helped |
| use the (p,p) variant instead of (p,2) (ablation) | Stacey on llama-100m · C4 pretraining test @30k steps | Testing Loss: 5.5755\std{0.0375} -> 4.4879\std{0.0132} | Helped |
| choose p=2 rather than other tested p values (ablation) | Stacey on ResNet-18 · CIFAR image classification | Testing accuracy | Helped |
| choose p=3 rather than other tested p values (ablation) | Stacey on llama-100m · C4 pretraining | Testing loss | Helped |
Surprising findings
- Stacey(p,2) has notably higher training loss than baselines on ImageNet and LLM pretraining despite its theory, and its ImageNet accuracy trails AdamW and Lion.
- The best p differs by task: p near 2 for CIFAR ResNet-18 but p around 3 for LLM pretraining.
Limitations
- Theory gives O(eps^-4) only for stochastic l_p steepest descent; the accelerated Stacey algorithm lacks a non-convex convergence guarantee in the stated results.
- Hyperparameters were tuned by grid search per optimizer, and experiments use modest batch sizes and limited scale (ResNet-18/50, llama-100m) due to compute limits.
- Assumes bounded gradients in the dual norm and unbiased, bounded-variance stochastic gradients.
Experimental setup
Datasets: CIFAR ImageNet C4 subset
Benchmarks: CIFAR test accuracy ImageNet top-1 accuracy C4 test loss
Metrics: training NLL test accuracy training loss test loss
Baselines: SGD with momentum Adam AdamW Lion
Builds on: signSGD Lion-K Schedule-Free SGD/AdamW linear coupling accelerated gradient descent AdamW
Closest papers by meaning
- Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness (NeurIPS 2025 · similarity 0.88)
- The Implicit Bias of Steepest Descent with Mini-batch Stochastic Gradient (ICML 2026 · similarity 0.87)
- Convergence of Steepest Descent and Adam under Non-Uniform Smoothness (ICML 2026 · similarity 0.87)
- Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.86)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.86)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.