Research map / Optimization / Optimizers
Lion Secretly Solves a Constrained Optimization: As Lyapunov Predicts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that Lion (and a Lion-phi family) is a principled method that solves a bound-constrained (or composite) optimization problem, proven via a new Lyapunov function in continuous and discrete time.
Problem
Lion, an optimizer discovered by program search, works well empirically but lacks theoretical grounding; it is unclear what optimization problem it solves and whether it converges.
Approach
Generalizes Lion to Lion-phi by replacing sign with a subgradient of a convex phi; shows via a new Lyapunov function that Lion-phi solves min f(x)+phi*(x) in continuous and discrete time, so Lion solves min f(x) s.t. ||x||_inf <= 1/lambda, where the weight decay sets the bound.
Key results
- Lyapunov function H(x,m) decreases monotonically along Lion-phi ODE trajectories, showing Lion-phi minimizes f(x)+phi*(x).
- Phase 1 shows distance from lambda*x_t to dom phi* decays exponentially, so the bound constraint is enforced quickly (weights confined within ~200 steps on ResNet-18 with lambda=10).
- Lion-phi with lambda=0.56 threshold: training loss increases only when weight decay excludes the unconstrained minimum.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Lion-ℓ1 instead of Lion-ℓp with other p | Lion on ViT-B/16 · ImageNet | training performance | Helped |
| Use Lion-ℓ1 instead of Lion-ℓp with other p | Lion on GPT-2 · OpenWebText language modeling | training performance | Helped |
| Use smaller p instead of larger p (ablation) | Lion-ℓp on ViT-B/16 and GPT-2 · ImageNet and OpenWebText language modeling | training efficiency | Helped |
| Use smaller p instead of larger p (ablation) | Lion-ℓp on ResNet-50 · ImageNet | training efficiency | Hurt |
| Increase weight decay λ (ablation) | Lion · CIFAR-10 | convergence speed | Helped |
| Increase weight decay beyond the optimal value (ablation) | Lion · CIFAR-10 | training loss: λ (=0.56) | Hurt |
Surprising findings
- Lion's sign-based update with decoupled weight decay solves a constrained problem rather than an l2-regularized one.
- Lion-l1 performs best among Lion-lp variants on ViT and GPT-2, while the trend reverses on ResNet-50.
Limitations
- Discrete-time analysis has an O(epsilon) accuracy term due to non-smoothness of phi.
- Focus is on theoretical framework; practical applications of Lion-phi variants left to future work.
- Mechanism of gradient enhancement and Lyapunov function derivation described as non-intuitive.
Experimental setup
Datasets: CIFAR-10 ImageNet OpenWebText
Benchmarks: CIFAR-10 ImageNet OpenWebText
Metrics: training loss parameter norm validation accuracy
Baselines: AdamW Polyak momentum Nesterov momentum signed gradient descent Hamiltonian descent
Builds on: Lion AdamW Hamiltonian descent Nesterov momentum Polyak momentum
Closest papers by meaning
- On PI Controllers for Updating Lagrange Multipliers in Constrained Optimization (ICML 2024 · similarity 0.86)
- When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$-norm Lower Bounds (NeurIPS 2026 · similarity 0.85)
- Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise (NeurIPS 2026 · similarity 0.85)
- Delve into the Applicability of Advanced Optimizers for Multi-Task Learning (NeurIPS 2026 · similarity 0.85)
- MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization (NeurIPS 2025 · similarity 0.85)
- How Memory in Optimization Algorithms Implicitly Modifies the Loss (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.