Research map / Continual learning / Continual fine-tuning of LLMs

Learning Dynamics in Continual Pre-Training for Large Language Models

ICML 2025 · Continual learning › Continual fine-tuning of LLMs · approach: Continual pretraining of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives a continual pre-training (CPT) scaling law that decomposes the CPT loss curve into an LR-annealing scaling law plus a power-law distribution shift term, predicting per-step general (D_pt) and downstream (D_cpt) validation loss and guiding hyper-parameter choice.

Problem

Existing scaling laws for continual pre-training only predict final loss and do not describe how general and downstream validation losses evolve at each CPT step under varying learning rate schedules, peak LRs, and replay ratios.

Approach

The CPT loss curve is a transfer curve between two hidden pre-training curves; it is modeled as an LR-annealing scaling law (forward and annealing areas) plus a power-law distribution shift term driven by the CPT forward area, enabling step-wise prediction and hyper-parameter optimization for balancing general and downstream loss.

Key results

• The fitted CPT scaling law reaches R^2 of 0.9944 on D_pt and 0.9993 on D_cpt validation loss for the main setting (Huber loss 0.0016 and 0.0021).
• Higher PT loss potential (less annealed PT model) consistently yields lower D_cpt validation loss after CPT.
• Out-of-domain validation loss can be predicted as a linear combination of D_pt and D_cpt losses, with nearly perfect prediction for several OOD datasets.

Measured effects reported in the paper

Surprising findings

• Performance degradation on the PT domain during CPT is inevitable if the turning length is infinitely large, unless the PT model is inadequately trained or the distribution shift is weak.
• The optimal replay ratio deviates from the target weight because the PT model was already trained on D_pt, producing a wavy pattern rather than a simple line.

Limitations

• Laws are primarily empirical with no rigorous theoretical derivation.
• Model sizes tested (106M to 1.7B non-embedding) are far below mainstream LLM scales, so model-size conclusions are assumptions.
• Open-source model adaptation relies on proxy PT data (RedPajama C4) and assumes final LR is zero.

Experimental setup

Datasets: FineWeb Knowledge-Pile Pile-of-Law RedPajama (C4 proxy) StackExchange arXiv PhilPapers Books SlimPajama Open-Web-Math

Benchmarks: FineWeb validation loss (D_pt) Knowledge-Pile validation loss (D_cpt) Pile-of-Law validation loss out-of-domain validation sets

Metrics: validation loss Huber loss R^2 absolute error

Baselines: D-CPT law CMR

Builds on: Chinchilla scaling law scaling law with LR annealing (WSD) D-CPT law Data Mixing Law

Closest papers by meaning

• D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models (NeurIPS 2024 · similarity 0.91)
• A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules (ICLR 2025 · similarity 0.89)
• Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling (ICLR 2026 · similarity 0.88)
• In-Place Test-Time Training (ICLR 2026 · similarity 0.88)
• Spurious Forgetting in Continual Learning of Language Models (ICLR 2025 · similarity 0.87)
• Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training (NeurIPS 2025 · similarity 0.87)