Research map / Optimization / Optimizers
Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling
Dmitrii Feoktistov, Timofey Belinsky, Andrey Veprikov, Amir Zainullin, Aleksandr Beznosikov
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace hard sign with smooth soft sign and temperature schedule | Signum · Next-character prediction (Transformer) | accuracy (%): 56.17 -> 57.05 | Improved |
| replace hard sign with smooth soft sign and temperature schedule | Signum · Next-character prediction (LSTM) | accuracy (%): 59.71 -> 61.36 | Improved |
| replace hard switch to SGD with smooth transition (ablation) | SoftSignum vs HardSwitchSign · Next-character prediction (LSTM) | accuracy (%): 60.40 -> 61.36 | Improved |
| apply soft spectral transition to Muon | Muon · Next-character prediction (Transformer) | accuracy (%): 57.36 -> 57.84 | Improved |
| apply soft spectral transition to Muon | Muon · Next-character prediction (LSTM) | accuracy (%): 63.02 -> 63.02 | No clear effect |
| apply soft spectral transition to Muon | Muon · GraphLand (Graph Transformer) | average rank: 2.43 -> 1.86 | Improved |
| replace hard sign with soft sign | Signum · GraphLand (Graph Transformer) | average rank: 3.86 -> 2.71 | Improved |
| apply soft spectral transition to Muon | Muon · LLM pretraining (720M, FineWeb) | perplexity: 16.362 -> 16.216 | Improved |
| decrease alpha_sign from 0.9 to 0.3 (ablation) | SoftSignum · Next-character prediction (Transformer) | val accuracy (%): 57.05 -> 57.20 | No clear effect |
| decrease alpha_sign from 0.9 to 0.3 (ablation) | SoftSignum · LLM pretraining (130M LLaMA, C4) | perplexity: 18.519 -> 18.559 | No clear effect |
| use SoftSignum instead of SGD under heavy class imbalance (k >= 10) | SoftSignum · Unbalanced CIFAR-10 | F1-score | Improved |
| use pure Signum instead of SoftSignum under extreme imbalance (k >= 100) | Signum · Unbalanced CIFAR-10 | F1-score | Hurt |
Closest papers by meaning
- When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$-norm Lower Bounds (NeurIPS 2026 · similarity 0.88)
- Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise (NeurIPS 2026 · similarity 0.87)
- Efficient Sign-Based Optimization: Accelerating Convergence via Variance Reduction (NeurIPS 2024 · similarity 0.87)
- Lion Secretly Solves a Constrained Optimization: As Lyapunov Predicts (ICLR 2024 · similarity 0.85)
- Exact risk curves of signSGD in High-Dimensions: quantifying preconditioning and noise-compression effects (ICML 2025 · similarity 0.84)
- Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
