AI Paper Atlas

Research map / Continual learning / Continual fine-tuning of LLMs

Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting

Ishaan Watts, Catherine Li, Sachin Goyal, Jacob Mitchell Springer, Aditi Raghunathan

ICML 2026 · Continual learning › Continual fine-tuning of LLMs · approach: Continual pretraining of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use SAM instead of AdamW in pretrainingOLMo-60M, 192B tokens · StarCoder fine-tuningforgetting at matched fine-tuning loss: +0.5 -> +0.1 pretraining loss degradation (80% less forgetting)Improved
scale pretraining tokens from 12B to 192B (ablation)SAM vs AdamW gap, OLMo-60M · StarCoder learning-forgetting frontierlearning-forgetting tradeoff gapImproved
use SAM at matched base pretraining loss (not just early stopping effect) (ablation)OLMo-60M · StarCoder, MusicPile, StackMathQApretraining loss after fine-tuningImproved
combine SAM-pretrained checkpoint with EWC during fine-tuningOLMo-60M · learning-forgetting frontierlearning-forgetting tradeoffImproved
increase peak pretraining learning rate beyond loss-optimal value (ablation)AdamW cosine, OLMo-60M 192B tokens · StarCoder fine-tuninglearning-forgetting frontierImproved
shorten WSD annealing period to 5% of training (ablation)WSD AdamW, OLMo-60M 192B tokens · StarCoder fine-tuninglearning-forgetting frontier: best at 5% vs 20% optimal for base lossImproved
use SAM instead of AdamW in pretrainingOLMo-60M · 24B-token 4-bit quantizationquantization-induced loss increase: 0.14 -> 0.08 (42% reduction)Improved
use SAM instead of AdamW at high token budgetsOLMo-60M · 4-bit quantizationquantization-induced loss increase: roughly 2-3x lessImproved
use SAM pretrainingOLMo-60M · isotropic Gaussian weight noisepretraining loss degradationImproved
switch to SAM only during the 10% WSD decay phase (ablation)WSD AdamW, OLMo-60M 192B tokens · StarCoder fine-tuning and 4-bit quantizationlearning-forgetting frontier: ~10% extra compute vs ~100% for full-run SAMImproved
SAM mid-training for 50B tokens instead of OLMo baseline annealingOLMo-2-1B · MetaMath post-trainingbenchmark degradation after post-training: 31% less forgettingImproved
SAM mid-training instead of OLMo baselineOLMo-2-1B · MusicPile post-trainingbenchmark degradation after post-trainingNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy