Research map / Architectures and efficiency / Pruning and efficient tuning
Symbolic Discovery of Optimization Algorithms
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, Quoc V Le
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace AdamW with Lion | ViT-B/16 trained from scratch · ImageNet | top-1 accuracy: 75.48 -> 77.44 | Improved |
| replace AdamW with Lion under strong augmentation | ViT-B/16 · ImageNet (RandAug+Mixup) | top-1 accuracy: 80.12 -> 80.77 | Improved |
| replace AdamW with Lion | ResNet-50 · ImageNet | top-1 accuracy: 76.34 -> 76.45 | No clear effect |
| replace Adafactor with Lion | BASIC-L · ImageNet zero-shot | top-1 accuracy: 85.7 -> 88.3 | Improved |
| replace AdamW with Lion | ViT-L/16 JFT-300M · ImageNet-A | accuracy: +6.08% | Improved |
| replace AdamW with Lion | U-Net diffusion · ImageNet 256x256 | FID: 4.7 -> 4.1 | Improved |
| replace Adafactor with Lion | 7.5B LM on 1.6T-token corpus · pretraining | perplexity | No clear effect |
| replace Adafactor with Lion | 1.1B LM · One-shot avg NLG | exact match: 11.1 -> 12.1 | Improved |
| replace AdamW with Lion | Imagen base 64x64 · MSCOCO text-to-image | FID/CLIP | No clear effect |
| replace AdamW with Lion | T5-11B · GLUE dev | average score: 91.08 -> 91.60 | Improved |
| use two interpolations (beta1, beta2) instead of single momentum EMA before sign (ablation) | sign-momentum optimizer · ImageNet ViT-B/16 | top-1 accuracy: 79.90 -> 80.77 | Improved |
| increase batch size to 32K (ablation) | Lion vs AdamW, ViT-B/16 · ImageNet | top-1 accuracy: 75.4 -> 77.9 | Improved |
Closest papers by meaning
- Lion Secretly Solves a Constrained Optimization: As Lyapunov Predicts (ICLR 2024 · similarity 0.88)
- SYMBOL: Generating Flexible Black-Box Optimizers through Symbolic Equation Learning (ICLR 2024 · similarity 0.87)
- Adam-mini: Use Fewer Learning Rates To Gain More (ICLR 2025 · similarity 0.87)
- MARS: Unleashing the Power of Variance Reduction for Training Large Models (ICML 2025 · similarity 0.87)
- How Memory in Optimization Algorithms Implicitly Modifies the Loss (NeurIPS 2025 · similarity 0.86)
- Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
