Research map / Continual learning / Continual fine-tuning of LLMs
What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Forecasts which upstream pretraining examples will be forgotten when a language model is refined on a single online-corrected error, using a logit-change-based interpretable model and a representation inner-product black-box model, and replays forecasted examples to reduce forgetting.
Problem
Updating a deployed language model with corrected error instances causes catastrophic forgetting of upstream examples, and random replay is inefficient and poorly controllable; the task is to forecast which upstream examples will be forgotten without costly inference.
Approach
Observes logit-change transfer from online-learned examples to upstream examples, approximates it with a trainable kernel over low-dimensional encoder representations (partially interpretable), and a black-box inner-product representation classifier with frequency priors that forecasts forgetting; forecasted examples are then replayed during refinement.
Key results
- Representation-based forecasting with frequency prior reaches 79.32 F1 on BART0-Large P3-Test with LM-head tuning, versus 62.96 for threshold-based.
- Sequential refinement of FLAN-T5-3B with LoRA: replaying forecasted-forgotten examples reduces EM drop from 4.384% (vanilla FT, FLAN-T5-Large) and 1.910% (random replay) to 0.138% for representation-based replay on FLAN-T5-3B.
- Trainable logit-based forecasting performs decently on BART0 but fails on FLAN-T5 models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use representation-based forgetting forecasts instead of threshold-based forecasts | Forgetting prediction · P3-Test, BART0-Large, Full FT | F1: 55.75 -> 67.19 | Helped |
| Use representation-based forgetting forecasts instead of threshold-based forecasts | Forgetting prediction · MMLU, FLAN-T5-Large, LoRA and Full FT | F1: 43.93 -> 48.66; 48.43 -> 51.51 | Helped |
| Use representation-based forgetting forecasts instead of threshold-based forecasts | Forgetting prediction · P3-Test OOD | F1: 46.24 -> 50.12 | Helped |
| Replace representation forecasting with fixed-logit forecasting | Forgetting prediction · MMLU, FLAN-T5-Large, Head | F1: 68.37 vs 67.81 | No clear effect |
| Use trainable logit forecasting instead of threshold forecasting | Forgetting prediction · P3-Test, BART0-Large, Full FT | F1: 55.75 -> 57.15 | Helped |
| Use trainable logit forecasting instead of threshold forecasting | Forgetting prediction · FLAN-T5 models with LoRA or Full FT | F1 | Hurt |
| Remove the frequency prior from representation-based forecasting (ablation) | Representation forecasting · P3-Test, BART0-Large, Head and Full FT | F1: 79.32 -> 77.92; 67.19 -> 66.53 | Hurt |
| Remove the frequency prior from representation-based forecasting (ablation) | Representation forecasting · MMLU, FLAN-T5-Large, LoRA and Full FT | F1: 48.66 -> 47.11; 51.51 -> 50.38 | Hurt |
| Replay examples forecasted to be forgotten instead of using vanilla fine-tuning | Sequential model refinement · P3-Test, BART0-Large, Full FT | EM Drop %: 9.274 -> 1.634 | Helped |
| Replay examples forecasted to be forgotten instead of using vanilla fine-tuning | Sequential model refinement · MMLU, FLAN-T5-Large, LoRA | EM Drop %: 5.463 -> 0.301 | Helped |
| Replay examples forecasted to be forgotten instead of using vanilla fine-tuning | Sequential model refinement · MMLU, FLAN-T5-Large, Full FT | EM Drop %: 3.302 -> 0.582 | Helped |
| Replay examples forecasted to be forgotten instead of using vanilla fine-tuning | Sequential model refinement · MMLU, FLAN-T5-3B, LoRA | EM Drop %: 4.384 -> 0.138 | Helped |
Surprising findings
- The logit-based forecasting model that works on BART0 fails on FLAN-T5 models, indicating logit-change transfer is not captured by the simplified kernel across model types.
- Increasing random replay batches reduces forgetting in multi-error streams but increases it in single-error settings.
Limitations
- Logit-based forecasting depends on model type, succeeding on BART0 but failing on FLAN-T5.
- Recall of forecasting drops over long sequential update streams.
- Forecasting models are not updated alongside the language model during continual refinement.
Experimental setup
Datasets: P3 MMLU SuperGLUE StoryCloze HellaSwag ANLI WinoGrande
Benchmarks: P3-Test MMLU
Metrics: F1 Edit Success Rate EM Drop Ratio precision recall
Baselines: Threshold-based forecasting Fixed logit forecasting Random replay MIR OCS MEND Vanilla FT
Builds on: neural tangent kernel MIR OCS MEND LoRA
Closest papers by meaning
- Demystifying Language Model Forgetting with Low-rank Example Associations (NeurIPS 2025 · similarity 0.92)
- Spurious Forgetting in Continual Learning of Language Models (ICLR 2025 · similarity 0.88)
- What to Forget in Unlearning? Forget Set Curation for Language Models (NeurIPS 2026 · similarity 0.87)
- Forgetting Has Neighbors: Localized Collateral Forgetting in Machine Unlearning (NeurIPS 2026 · similarity 0.87)
- Learning and Forgetting Unsafe Examples in Large Language Models (ICML 2024 · similarity 0.87)
- How Much Can We Forget about Data Contamination? (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.