Research map / Large language models / Instruction and fine-tuning
COSMOS: A Hybrid Adaptive Optimizer for Efficient Training of Large Language Models
Liming Liu, Zhenghao Xu, Zixuan Zhang, Hao Kang, Zichong Li, Chen Liang, Weizhu Chen, Tuo Zhao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace MUON with COSMOS (SOAP on leading subspace, MUON on residual) | MUON · C4 LLaMA-130M | validation perplexity: 20.69 -> 20.54 | Improved |
| replace MUON with COSMOS at 350M | MUON · C4 LLaMA-350M | validation perplexity: 16.49 -> 16.21 | Improved |
| replace MUON with COSMOS at 1B | MUON · C4 LLaMA-1B | validation perplexity: 12.57 -> 12.46 | Improved |
| COSMOS versus SOAP at 350M with far less memory | SOAP · C4 LLaMA-350M | validation perplexity: 16.32 -> 16.21 | Improved |
| COSMOS instead of SOAP for 1B memory | SOAP · LLaMA-1B max GPU memory | max GPU memory: 72.58 G -> 58.47 G | Improved |
| COSMOS instead of SOAP for throughput | SOAP · LLaMA-1B A100 | throughput: 6.33 -> 7.07 | Improved |
| COSMOS instead of MUON for memory/time | MUON · LLaMA-1B profiling | wall-clock time: 35.56 s -> 35.75 s | No clear effect |
| use COSMOS instead of MUON at learning rate 2e-3 (ablation) | MUON · C4 LLaMA-130M | validation perplexity: 26.74 -> 21.00 | Improved |
| fix gamma to 0.25 while varying learning rate to 2e-3 (ablation) | COSMOS · C4 LLaMA-130M | validation perplexity: 21.00 -> 20.87 | Improved |
| increase rank from 32 to 128 at gamma=1 (ablation) | COSMOS · C4 LLaMA-130M | validation perplexity: 20.54 -> 20.72 | Hurt |
| vary discount factor gamma from 0.1 to 0.25 at rank 64 (ablation) | COSMOS · C4 LLaMA-130M | validation perplexity: 20.62 -> 20.54 | Improved |
| add normalization to MUON (ablation) | MUON · C4 LLaMA-130M and 350M | validation loss | No clear effect |
Closest papers by meaning
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.89)
- Search for Efficient Large Language Models (NeurIPS 2024 · similarity 0.89)
- Second-Order Fine-Tuning without Pain for LLMs: A Hessian Informed Zeroth-Order Optimizer (ICLR 2025 · similarity 0.88)
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.88)
- APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference (ICML 2024 · similarity 0.88)
- LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
