AI Paper Atlas

Research map / Continual learning / Continual fine-tuning of LLMs

Can Muon Fine-tune Adam-Pretrained Models?

Xingyu Qu, Huang Peigeng, Samuel Horváth

ICML 2026 · Continual learning › Continual fine-tuning of LLMs · approach: Synthetic data and merging for CLIP · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
fine-tune with Muon instead of Adam on Adam-pretrained model (full fine-tuning, mismatch) (ablation)NanoChat 561M Adam-pretrained · WikiText-2 val perplexityrelative perplexity: 0.710 -> 0.719Hurt
fine-tune with Adam instead of Muon on Muon-pretrained model (full fine-tuning, mismatch) (ablation)NanoChat 561M Muon-pretrained · WikiText-2 val perplexityrelative perplexity: 0.716 -> 0.739Hurt
use LoRA instead of full fine-tuning with mismatched optimizer (ablation)Muon fine-tuning of Adam-pretrained NanoChat · WikiText-2 val perplexitymismatch gap in relative perplexity: +.009 -> +.002Improved
use LoRA instead of full fine-tuning with Muon (ablation)Muon fine-tuning of Llama 2-7B · GSM8Kaccuracy %: 57.37 -> 59.57 (gap to Adam 4.29 -> 2.09 narrowed)Improved
use LoRA with Muon instead of full fine-tuning on T5-Base GLUE (ablation)Muon · GLUE averageaccuracy %: 88.77 -> 88.97Improved
use Polar Express coefficients in Muon with LoRA (T5-Base GLUE) (ablation)LoRA-Muon · GLUE averageaccuracy %: 88.97 -> 89.20Improved
use Polar Express coefficients in Muon (Llama 2-7B) (ablation)Muon · Llama 2-7B instruction tuning averageaverage: Full 53.10 -> 53.56; LoRA 52.15 -> 51.82Mixed
increase LoRA rank for Muon on MetaMath (ablation)LoRA-Muon · GSM8K (Llama 2-7B)accuracy %Hurt
vary LoRA rank on Code-Feedback (ablation)LoRA-Muon · HumanEval (Llama 2-7B)Pass@1 %No clear effect
use LoRA instead of full fine-tuning for Muon (forgetting) (ablation)Muon · Commonsense after MetaMath fine-tuningaverage accuracy %: 55.4 -> 56.9Improved
add rsLoRA scaling to LoRA-Muon-PE (ablation)LoRA-Muon-PE · GLUE average (T5-Base)accuracy %: 89.20 -> 89.12Hurt
add LoRA-One initialization to LoRA-Muon-PE (ablation)LoRA-Muon-PE · GLUE average (T5-Base)accuracy %: 89.20 -> 89.09Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy