Research map / Optimization / Optimizers
Sign-SGD via Parameter-Free Optimization
Daniil Medyakov, Stanko Sergey, Gleb Molodtsov, Philip Zmushko, Evseev Grigoriy, Egor Petrov, Aleksandr Beznosikov
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace tuned cosine-scheduler stepsize with parameter-free ALIAS (no momentum) | Sign-SGD · C4 LLaMA-130M | validation loss: 2.980 -> 3.006 (with wd) | No clear effect |
| replace tuned AdamW and Prodigy with ALIAS Adam version | ALIAS Adam · C4 LLaMA-130M | validation loss: 2.929 (AdamW) -> 2.918 | Improved |
| use ALIAS Adam version instead of other parameter-free optimizers | ALIAS Adam · C4 LLaMA-130M | validation loss: 2.925 (MoMo) -> 2.918 | Improved |
| add momentum to ALIAS (Adam version) | ALIAS · C4 LLaMA-130M | validation loss: 3.006 -> 2.976 | Improved |
| add cosine scheduler on top of ALIAS Adam version (ablation) | ALIAS Adam · C4 LLaMA-130M | validation loss: 2.976 -> 2.918 | Improved |
| use ALIAS Adam version on 350M model | ALIAS Adam · C4 LLaMA-350M | validation loss: 2.717 (Sign-SGD) / 2.719 (AdamW) -> 2.707 | Improved |
| use approximated l_inf norm in memory-efficient ALIAS (ablation) | ALIAS · C4 LLaMA-130M | validation loss | Hurt |
| add constant L_inf term to ALIAS stepsize (0 to 1000) (ablation) | ALIAS · C4 LLaMA-130M | validation loss: 3.006 -> 3.006 (L_inf 0 vs 1000) | No clear effect |
| change initial distance d0 from 1 to 10^-3 (ablation) | ALIAS Adam · C4 LLaMA-130M | validation loss: 2.918 -> 2.918 | No clear effect |
| reduce batch size from 512 to 64 (more gradient noise) (ablation) | ALIAS · C4 LLaMA-130M | validation loss: 3.006 -> 3.029 (Sign-SGD 2.980 -> 2.999) | Hurt |
| use ALIAS Adam version for Swin fine-tuning instead of tuned AdamW | ALIAS Adam · Tiny ImageNet (Swin-T) | final accuracy: 77.612 -> 79.161 | Improved |
| use ALIAS Adam version instead of AdamW on OGBG | ALIAS Adam · AlgoPerf OGBG | mAP: 0.254 -> 0.242 | Hurt |
Closest papers by meaning
- Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise (NeurIPS 2026 · similarity 0.89)
- When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$-norm Lower Bounds (NeurIPS 2026 · similarity 0.88)
- SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training (ICML 2025 · similarity 0.87)
- Efficient Sign-Based Optimization: Accelerating Convergence via Variance Reduction (NeurIPS 2024 · similarity 0.87)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.86)
- z-SignFedAvg: A Unified Stochastic Sign-Based Compression for Federated Learning (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
