AI Paper Atlas

Research map / Large language models / Instruction and fine-tuning

COSMOS: A Hybrid Adaptive Optimizer for Efficient Training of Large Language Models

Liming Liu, Zhenghao Xu, Zixuan Zhang, Hao Kang, Zichong Li, Chen Liang, Weizhu Chen, Tuo Zhao

ICLR 2026 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace MUON with COSMOS (SOAP on leading subspace, MUON on residual)MUON · C4 LLaMA-130Mvalidation perplexity: 20.69 -> 20.54Improved
replace MUON with COSMOS at 350MMUON · C4 LLaMA-350Mvalidation perplexity: 16.49 -> 16.21Improved
replace MUON with COSMOS at 1BMUON · C4 LLaMA-1Bvalidation perplexity: 12.57 -> 12.46Improved
COSMOS versus SOAP at 350M with far less memorySOAP · C4 LLaMA-350Mvalidation perplexity: 16.32 -> 16.21Improved
COSMOS instead of SOAP for 1B memorySOAP · LLaMA-1B max GPU memorymax GPU memory: 72.58 G -> 58.47 GImproved
COSMOS instead of SOAP for throughputSOAP · LLaMA-1B A100throughput: 6.33 -> 7.07Improved
COSMOS instead of MUON for memory/timeMUON · LLaMA-1B profilingwall-clock time: 35.56 s -> 35.75 sNo clear effect
use COSMOS instead of MUON at learning rate 2e-3 (ablation)MUON · C4 LLaMA-130Mvalidation perplexity: 26.74 -> 21.00Improved
fix gamma to 0.25 while varying learning rate to 2e-3 (ablation)COSMOS · C4 LLaMA-130Mvalidation perplexity: 21.00 -> 20.87Improved
increase rank from 32 to 128 at gamma=1 (ablation)COSMOS · C4 LLaMA-130Mvalidation perplexity: 20.54 -> 20.72Hurt
vary discount factor gamma from 0.1 to 0.25 at rank 64 (ablation)COSMOS · C4 LLaMA-130Mvalidation perplexity: 20.62 -> 20.54Improved
add normalization to MUON (ablation)MUON · C4 LLaMA-130M and 350Mvalidation lossNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy