Research map / Continual learning / Continual fine-tuning of LLMs

Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition

ICLR 2025 · Continual learning › Continual fine-tuning of LLMs · approach: Continual pretraining of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Defines knowledge entropy over FFN memory coefficients in OLMo checkpoints, shows it declines during pretraining, and links this decline to worse knowledge acquisition and forgetting in continual learning, partly reversed by resuscitating low-activation memory vectors.

Problem

How a language model's breadth of integrating parametric knowledge changes across pretraining, and how this affects knowledge acquisition and forgetting when continually trained on new data.

Approach

Defines knowledge entropy as the entropy of normalized average FFN memory coefficients (absolute SwiGLU activations times value-memory rows) summed over layers, and shows it falls over pretraining; then artificially raises low coefficients in the key projection to test whether restoring activity improves acquisition and retention.

Key results

• Knowledge entropy of OLMo consistently decreases over pretraining in both 1B and 7B models, with the largest drop in the last layer.
• Knowledge entropy correlates with knowledge acquisition (Pearson 0.94) and forgetting (Pearson -0.96) across OLMo 1B checkpoints.
• Resuscitating the lowest 50% of memory coefficients in final-stage OLMo-1B (q>=1) improves acquisition and retention over the unmodified final checkpoint, though it remains below the mid-stage model.

Measured effects reported in the paper

Surprising findings

• Resuscitating inactive memory vectors in final-stage models improves acquisition and retention, while the same intervention hurts early and mid-stage models.
• Acquisition rate for the final-stage OLMo model is negative in the once setting, meaning injected knowledge is forgotten during continual learning.

Limitations

• Only OLMo 1B and 7B evaluated because they are the only models with public intermediate checkpoints; Pythia analysis deviates from the trend.
• Resuscitation is an arbitrary parameter scaling that does not fully restore plasticity and can hurt early/mid-stage models.
• Knowledge acquisition and forgetting are measured only in a continual learning setup, not during pretraining itself.

Experimental setup

Datasets: Dolma PubMed C4 Fictional Knowledge SciQ Winogrande PIQA OBQA HellaSwag ARC Easy

Benchmarks: Fictional Knowledge SciQ Winogrande PIQA OBQA HellaSwag ARC Easy

Metrics: knowledge acquisition A(theta) knowledge forgetting F(theta) probe performance K(theta) average downstream accuracy P(theta) knowledge entropy Pearson correlation

Baselines: OLMo intermediate checkpoints at early, mid, and final stages Pythia 1.4B

Builds on: key-value memory in feed-forward layers continual knowledge learning Fictional Knowledge dataset

Closest papers by meaning

• How Do Large Language Models Acquire Factual Knowledge During Pretraining? (NeurIPS 2024 · similarity 0.90)
• Spurious Forgetting in Continual Learning of Language Models (ICLR 2025 · similarity 0.89)
• How new data permeates LLM knowledge and how to dilute it (ICLR 2025 · similarity 0.88)
• The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations (NeurIPS 2026 · similarity 0.88)
• Enhancing Elusive Clues in Knowledge Learning by Contrasting Attention of Language Models (AAAI 2025 · similarity 0.88)
• Data Mixing Can Induce Phase Transitions in Knowledge Acquisition (NeurIPS 2025 · similarity 0.87)