Research map / Continual learning / Domain and prompt-based
Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives and validates scaling laws predicting pretraining-loss forgetting and finetuning loss as a function of model size, unique finetuning tokens, and the fraction of pretraining data injected into the finetuning mix.
Problem
Finetuning a pretrained LM on limited target-domain data causes overfitting and forgetting of pretraining knowledge, and it is unclear how much pretraining data to mix in to mitigate these effects.
Approach
A multiplicative scaling law for post-finetuning pretraining loss that treats a fraction p of injected pretraining data as allocating (1+Bp)N effective parameters, combined with a finetuning-loss law that is approximately independent of p; finds that as little as 1% pretraining injection prevents forgetting.
Key results
- Forgetting scaling law predicts pretraining loss after finetuning with a bootstrapped mean relative error of 0.40% averaged across domains, versus 0.89% for the finetuning-loss law.
- Injecting as little as p=1% of pretraining data prevents forgetting of the pretraining set at no cost to finetuning performance.
- Smaller models lose up to 95% of pretraining progress when forgetting, while larger models lose about 20%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Inject 1% pretraining data during finetuning rather than injecting none | Full-parameter finetuning · Pretraining set and target domains | pretraining loss after finetuning | Helped |
| Injecting pretraining data has little effect on target-domain loss | Full-parameter finetuning · Pile domains | finetuning validation loss | No clear effect |
| Use a multiplicative rather than additive finetuning scaling law (ablation) | Finetuning scaling-law prediction · 12 Pile domains | finetuning MRE: 1.36% -> 0.89% | Helped |
| Include the base pretraining loss in the forgetting scaling law (ablation) | Proposed forgetting scaling law · 12 Pile domains | forgetting MRE: 1.05% -> 0.40% | Helped |
| Account for the rewarmed pretraining loss when estimating the base loss (ablation) | Proposed forgetting scaling law · 12 Pile domains | forgetting MRE: 0.49% -> 0.40% | Helped |
| Replace the proposed forgetting law with A D^β(1-p)^κ/N^α + E (ablation) | Forgetting scaling-law prediction · 12 Pile domains | forgetting MRE: 0.40% -> 0.67% | Hurt |
| Use pretraining-data mixing instead of Anchored AdamW parameter regularization (ablation) | Finetuning on Github · Github and RedPajama | loss | Mixed |
Surprising findings
- Only 0.3 unique pretraining tokens per unique finetuning token are sufficient to avoid forgetting.
- Pretraining data injection can lower the finetuning validation loss at the bottom of the U-curve, acting as a regularizer.
Limitations
- Only full-parameter finetuning studied; PEFT/LoRA deferred to future work.
- Pretraining done along isocurve D=100N only, not the full (N, D) space.
- Constant learning rate finetuning with fixed 12K steps; results tied to these choices.
Experimental setup
Datasets: RedPajama-V2 The Pile (12 domains, e.g., ArXiv, DM Mathematics, Enron emails, GitHub, FreeLaw, Wikipedia en, OpenWebText2, PubMed) OpenHermes
Benchmarks: ARC-easy MMLU
Metrics: validation loss training loss mean relative error (bootstrapped) accuracy
Baselines: Additive scaling law Purely additive forgetting law Anchored AdamW AdamW weight decay finetuning
Builds on: Chinchilla scaling laws Scaling data-constrained language models continual pretraining with replay
Closest papers by meaning
- How Much Can We Forget about Data Contamination? (ICML 2025 · similarity 0.89)
- Scaling Laws for Precision (ICLR 2025 · similarity 0.88)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.88)
- Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws (NeurIPS 2026 · similarity 0.88)
- Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting (ICML 2025 · similarity 0.88)
- Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.