Research map / Continual learning / Online and replay-based
Reawakening knowledge: Anticipatory recovery from catastrophic interference via structured training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that LLMs (Pythia) fine-tuned cyclically on a fixed repeated document sequence recover from catastrophic forgetting on a document before it reappears ("anticipatory recovery"), an effect that emerges with model scale and also appears in some vision models, and offers a toy linear model and weight/gradient/activation analyses as partial explanation.
Problem
Neural networks, including LLMs, usually suffer catastrophic interference when trained sequentially on documents, but it is unclear how they behave when documents are presented cyclically in a fixed repeated order, as in naturalistic environments.
Approach
In cyclic fine-tuning with a fixed document order and multiple gradient steps per document, over-parameterized networks reverse forgetting before a document recurs; the effect is emergent with width/depth/scale and is explained through temporal structure in gradients, weight residuals, activations, and a toy linear model.
Key results
- Cyclic fine-tuning of pre-trained Pythia-1B recovers over half of initial forgetting on document 1 before revisiting it in epoch 2, with loss peaking around document 60.
- Average online loss across epochs 2-5 is lower for cyclic than shuffled fine-tuning at all four Pythia sizes (e.g., 1.03 vs 1.51 at 1B).
- Anticipatory recovery is emergent with scale: models need sufficient width (at least 512) and depth (at least 8 blocks) to show noticeable recovery.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a fixed document order across epochs rather than random shuffling | Pythia models fine-tuned on CNN/Daily Mail · CNN/Daily Mail prequential (epochs 2–5) | average online loss: 1.34 $\pm$ 0.02 -> 1.09 $\pm$ 0.03; 1.51 $\pm$ 0.04 -> 1.03 $\pm$ 0.03; 1.51 $\pm$ 0.04 -> 1.14 $\pm$ 0.06; 1.79 $\pm$ 0 | Helped |
| Increase model size (ablation) | Pre-trained Pythia models under cyclic fine-tuning · CNN/Daily Mail cyclic fine-tuning | anticipatory recovery score | Helped |
| Use a Pythia model initialization instead of isotropic Gaussian initialization (ablation) | Pythia-1B trained from random initialization · CNN/Daily Mail cyclic fine-tuning | anticipatory recovery | Helped |
| Increase the number of pre-training steps (ablation) | Pythia models · CNN/Daily Mail cyclic fine-tuning | anticipatory recovery | Helped |
| Use Adam instead of vanilla gradient descent (ablation) | Randomly initialized and pre-trained models · Cyclic fine-tuning | anticipatory recovery | Helped |
| Increase gradient steps per document (ablation) | Pythia models · Cyclic fine-tuning | recovery score | Helped |
| Divide the same total update among more gradient steps by scaling the learning rate inversely with the step count (ablation) | Pythia models · Cyclic fine-tuning | anticipatory recovery | Helped |
| Increase context length (ablation) | Pythia models · Cyclic fine-tuning | recovery score | Hurt |
| Freeze more transformer blocks (ablation) | Pre-trained Pythia-1B · Cyclic fine-tuning | recovery score | Hurt |
| Add variation to training data using random token masking or random context-window shifts (ablation) | Pythia models · Cyclic fine-tuning | anticipatory recovery | Hurt |
| Increase the number of attention heads (ablation) | Pythia-1B with width 2048 and depth 16 · Cyclic fine-tuning | anticipatory recovery | No clear effect |
| Use one gradient step per document at learning rate 0.001 (ablation) | Pythia models · Cyclic fine-tuning | anticipatory recovery | Hurt |
Surprising findings
- Recovery on a document occurs before it is seen again even though documents never appear together in context.
- Gradient similarity peaks near the center of the cycle rather than along the diagonal, indicating dependence on position in the cycle.
Limitations
- Setup is highly simplified (single repeated sequence of news articles); ablations run in a single setting.
- Toy model does not explain why the effect is stronger for LLMs/autoregressive tasks than other architectures.
- Theoretical foundations are not established; no formal theorems.
Experimental setup
Datasets: CNN/Daily Mail The Pile CIFAR-10 ImageNet wikitext-103
Benchmarks: CNN/Daily Mail CIFAR-10 ImageNet wikitext-103
Metrics: average cross-entropy loss recovery score prequential online loss
Baselines: random shuffled fine-tuning
Builds on: Pythia Image GPT Adam
Closest papers by meaning
- Layerwise Proximal Replay: A Proximal Point Method for Online Continual Learning (ICML 2024 · similarity 0.85)
- Memorization Sinks: Isolating Memorization during LLM Training (ICML 2025 · similarity 0.85)
- The Importance of Being Lazy: Scaling Limits of Continual Learning (ICML 2025 · similarity 0.85)
- Learning dynamics in linear recurrent neural networks (ICML 2025 · similarity 0.85)
- Learning without Isolation: Pathway Protection for Continual Learning (ICML 2025 · similarity 0.85)
- Kolmogorov-Arnold Networks Still Catastrophically Forget but Differently from MLP (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.