Research map / Continual learning / Continual fine-tuning of LLMs
Can Muon Fine-tune Adam-Pretrained Models?
Xingyu Qu, Huang Peigeng, Samuel Horváth
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with Muon instead of Adam on Adam-pretrained model (full fine-tuning, mismatch) (ablation) | NanoChat 561M Adam-pretrained · WikiText-2 val perplexity | relative perplexity: 0.710 -> 0.719 | Hurt |
| fine-tune with Adam instead of Muon on Muon-pretrained model (full fine-tuning, mismatch) (ablation) | NanoChat 561M Muon-pretrained · WikiText-2 val perplexity | relative perplexity: 0.716 -> 0.739 | Hurt |
| use LoRA instead of full fine-tuning with mismatched optimizer (ablation) | Muon fine-tuning of Adam-pretrained NanoChat · WikiText-2 val perplexity | mismatch gap in relative perplexity: +.009 -> +.002 | Improved |
| use LoRA instead of full fine-tuning with Muon (ablation) | Muon fine-tuning of Llama 2-7B · GSM8K | accuracy %: 57.37 -> 59.57 (gap to Adam 4.29 -> 2.09 narrowed) | Improved |
| use LoRA with Muon instead of full fine-tuning on T5-Base GLUE (ablation) | Muon · GLUE average | accuracy %: 88.77 -> 88.97 | Improved |
| use Polar Express coefficients in Muon with LoRA (T5-Base GLUE) (ablation) | LoRA-Muon · GLUE average | accuracy %: 88.97 -> 89.20 | Improved |
| use Polar Express coefficients in Muon (Llama 2-7B) (ablation) | Muon · Llama 2-7B instruction tuning average | average: Full 53.10 -> 53.56; LoRA 52.15 -> 51.82 | Mixed |
| increase LoRA rank for Muon on MetaMath (ablation) | LoRA-Muon · GSM8K (Llama 2-7B) | accuracy % | Hurt |
| vary LoRA rank on Code-Feedback (ablation) | LoRA-Muon · HumanEval (Llama 2-7B) | Pass@1 % | No clear effect |
| use LoRA instead of full fine-tuning for Muon (forgetting) (ablation) | Muon · Commonsense after MetaMath fine-tuning | average accuracy %: 55.4 -> 56.9 | Improved |
| add rsLoRA scaling to LoRA-Muon-PE (ablation) | LoRA-Muon-PE · GLUE average (T5-Base) | accuracy %: 89.20 -> 89.12 | Hurt |
| add LoRA-One initialization to LoRA-Muon-PE (ablation) | LoRA-Muon-PE · GLUE average (T5-Base) | accuracy %: 89.20 -> 89.09 | Hurt |
Closest papers by meaning
- LoRA vs Full Fine-tuning: An Illusion of Equivalence (NeurIPS 2025 · similarity 0.87)
- Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective (NeurIPS 2026 · similarity 0.86)
- Overtrained Language Models Are Harder to Fine-Tune (ICML 2025 · similarity 0.85)
- SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning (ICML 2025 · similarity 0.84)
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.84)
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
