Research map / Architectures and efficiency / Pruning and efficient tuning
BAS: Bridging Adam and SignSGD for Memory-Efficient LLM Training
Yijie Zhou, Mingliang Zhang, Jiaqi Zhang, Xunliang Cai, Shi Pu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use BAS instead of AdamW | Olmo3-7B SFT · MATH | score: 58.6 -> 59.2 | No clear effect |
| use BAS instead of AdamW | Olmo3-7B SFT · IFEval | score: 76.0 -> 75.2 | No clear effect |
| use BAS instead of AdamW | Olmo3-32B SFT · IFEval | score: 85.2 -> 84.1 | No clear effect |
| use BAS instead of AdamW | Olmo3-32B SFT · BigBenchHard | score: 66.7 -> 67.9 | No clear effect |
| use BAS instead of Lion | Olmo3-7B SFT · MATH | score: 56.5 -> 59.2 | Improved |
| use BAS instead of Adam-mini | Olmo3-7B SFT · HumanEval+ | score: 63.2 -> 63.7 | No clear effect |
| use BAS instead of AdamS | Olmo3-7B SFT · AIME 2024 | score: 5.4 -> 4.9 | No clear effect |
| use AdamW hyperparameters with BAS in place of Adam-mini and AdamS | Qwen2.5 pre-training (0.5B and 1.5B) · training loss | stability | Improved |
| compare Lion to BAS in pre-training | Qwen2.5 pre-training · training loss | convergence speed | Hurt |
| store first moment in FP8 instead of BF16 or FP32 (ablation) | BAS · pre-training loss | loss | No clear effect |
| store first moment in 6-bit instead of FP8 (ablation) | BAS · pre-training loss | loss | Hurt |
| use spectral norm instead of Frobenius norm (ablation) | BAS · convergence | loss | Hurt |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- Adam-mini — Adam-mini: Use Fewer Learning Rates To Gain More (ICLR 2025) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics (NeurIPS 2026 · similarity 0.88)
- Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise (ICLR 2025 · similarity 0.87)
- Exact risk curves of signSGD in High-Dimensions: quantifying preconditioning and noise-compression effects (ICML 2025 · similarity 0.87)
- In Search of Adam’s Secret Sauce (NeurIPS 2025 · similarity 0.87)
- SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training (ICLR 2025 · similarity 0.86)
- Adam-mini: Use Fewer Learning Rates To Gain More (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
