Research map / Architectures and efficiency / Pruning and efficient tuning
OLion: Approaching the Hadamard Ideal by Intersecting Spectral and L inf Implicit Biases
Zixiao Wang, Yifei Shen, Huishuai Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply sign after Newton-Schulz orthogonalization (OLion) instead of Muon orthogonalization alone | Llama-3.1-8B SFT from AdamW-pretrained checkpoint · GSM8K 0-shot | accuracy (%): 57.24 -> 60.04 | Improved |
| apply sign after orthogonalization (OLion) vs Muon | Llama-3.1-8B SFT · Aqua 0-shot | accuracy (%): 41.34 -> 46.45 | Improved |
| OLion vs AdamW for SFT of an AdamW-pretrained model | Llama-3.1-8B SFT · GSM8K 4-shot | accuracy (%): 60.35 -> 60.58 | No clear effect |
| OLion vs Muon on MATH 0-shot SFT | Llama-3.1-8B SFT · MATH 0-shot | accuracy (%): 19.32 -> 19.80 | No clear effect |
| use OLion instead of AdamW, Lion and Muon for GPT-2 pretraining | GPT-2 124M-770M on OpenWebText · validation loss | convergence speed | Improved |
| use OLion for Llama-2-7B pretraining | Llama-2-7B · training and validation loss | loss | Improved |
| vary learning rate (3e-4 to 5e-3) with OLion (ablation) | GPT-2 small · validation loss at step 10,000 | validation loss | Improved |
| use OLion instead of Muon for SiT-B/2 pretraining | SiT-B/2 on ImageNet-1K · projection and denoising loss | loss | Improved |
| add sign step after orthogonalization | GPT-2 small weights · spectral and l-inf norms | l-inf norm | Improved |
Closest papers by meaning
- Tensorion: A Tensor-Aware Generalization of the Muon Optimizer (NeurIPS 2026 · similarity 0.86)
- Delve into the Applicability of Advanced Optimizers for Multi-Task Learning (NeurIPS 2026 · similarity 0.83)
- Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds (NeurIPS 2026 · similarity 0.83)
- Lion Secretly Solves a Constrained Optimization: As Lyapunov Predicts (ICLR 2024 · similarity 0.83)
- LION: Implicit Vision Prompt Tuning (AAAI 2024 · similarity 0.83)
- Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
