Research map / Continual learning / Continual fine-tuning of LLMs
Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting
Ishaan Watts, Catherine Li, Sachin Goyal, Jacob Mitchell Springer, Aditi Raghunathan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SAM instead of AdamW in pretraining | OLMo-60M, 192B tokens · StarCoder fine-tuning | forgetting at matched fine-tuning loss: +0.5 -> +0.1 pretraining loss degradation (80% less forgetting) | Improved |
| scale pretraining tokens from 12B to 192B (ablation) | SAM vs AdamW gap, OLMo-60M · StarCoder learning-forgetting frontier | learning-forgetting tradeoff gap | Improved |
| use SAM at matched base pretraining loss (not just early stopping effect) (ablation) | OLMo-60M · StarCoder, MusicPile, StackMathQA | pretraining loss after fine-tuning | Improved |
| combine SAM-pretrained checkpoint with EWC during fine-tuning | OLMo-60M · learning-forgetting frontier | learning-forgetting tradeoff | Improved |
| increase peak pretraining learning rate beyond loss-optimal value (ablation) | AdamW cosine, OLMo-60M 192B tokens · StarCoder fine-tuning | learning-forgetting frontier | Improved |
| shorten WSD annealing period to 5% of training (ablation) | WSD AdamW, OLMo-60M 192B tokens · StarCoder fine-tuning | learning-forgetting frontier: best at 5% vs 20% optimal for base loss | Improved |
| use SAM instead of AdamW in pretraining | OLMo-60M · 24B-token 4-bit quantization | quantization-induced loss increase: 0.14 -> 0.08 (42% reduction) | Improved |
| use SAM instead of AdamW at high token budgets | OLMo-60M · 4-bit quantization | quantization-induced loss increase: roughly 2-3x less | Improved |
| use SAM pretraining | OLMo-60M · isotropic Gaussian weight noise | pretraining loss degradation | Improved |
| switch to SAM only during the 10% WSD decay phase (ablation) | WSD AdamW, OLMo-60M 192B tokens · StarCoder fine-tuning and 4-bit quantization | learning-forgetting frontier: ~10% extra compute vs ~100% for full-run SAM | Improved |
| SAM mid-training for 50B tokens instead of OLMo baseline annealing | OLMo-2-1B · MetaMath post-training | benchmark degradation after post-training: 31% less forgetting | Improved |
| SAM mid-training instead of OLMo baseline | OLMo-2-1B · MusicPile post-training | benchmark degradation after post-training | No clear effect |
Closest papers by meaning
- Forget Sharpness: Perturbed Forgetting of Model Biases Within SAM Dynamics (ICML 2024 · similarity 0.90)
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training (ICLR 2025 · similarity 0.89)
- ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think (ICML 2025 · similarity 0.88)
- Make Continual Learning Stronger via C-Flat (NeurIPS 2024 · similarity 0.88)
- Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting (ICML 2025 · similarity 0.88)
- Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
