Research map / Large language models / Instruction and fine-tuning
Demystifying Language Model Forgetting with Low-rank Example Associations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that LLM forgetting matrices (learned tasks x upstream examples) are well approximated by low-rank structure, and uses matrix completion from 30 seed examples to predict forgetting and guide targeted replay.
Problem
Forgetting of upstream examples in LLMs after fine-tuning on new tasks is costly to measure exhaustively; it is unclear how forgotten examples depend on the newly learned task, and how to predict and efficiently mitigate it.
Approach
Treat the task-by-example forgetting matrix as approximately low-rank and predict forgetting on unseen tasks via matrix completion (additive, matrix factorization, KNN) from a few seed examples, then upweight predicted-forgotten examples for replay.
Key results
- Rank-3 approximation fits the OLMo-1B/7B association matrices with R^2 above 0.69, and rank-1 already exceeds 0.5.
- MF achieves the best in-domain forgetting prediction for OLMo-7B-Instruct with F1 58.16 versus 55.46 for SBERT, and the best RMSE for OLMo-1B and OLMo-7B in-domain.
- Replaying examples upweighted by offline-predicted forgetting (MF/KNN/additive) statistically significantly reduces log perplexity increase versus random replay (p<0.05) in several setups.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use matrix factorization on forgetting statistics instead of sentence-embedding prediction | Forgetting prediction · In-domain MPT | RMSE: 13.86 -> 10.41 | Helped |
| Use matrix factorization on forgetting statistics instead of sentence-embedding prediction | Forgetting prediction · In-domain OLMo-7B | RMSE: 7.44 -> 7.14 | Helped |
| Use matrix factorization on forgetting statistics instead of sentence-embedding prediction | Forgetting prediction · In-domain OLMo-7B-Instruct | F1: 55.46 -> 58.16 | Helped |
| Use matrix factorization on forgetting statistics instead of sentence-embedding prediction | Forgetting prediction · Out-of-domain MPT | RMSE: 10.59 -> 7.03 | Helped |
| Use matrix factorization on forgetting statistics instead of sentence-embedding prediction | Forgetting prediction · Out-of-domain OLMo-7B-Instruct | F1: 42.95 -> 42.91 | No clear effect |
| Increase MF rank from 1 to 8 (ablation) | Forgetting prediction with MF · OLMo-7B | RMSE: 7.31 -> 7.12 | Helped |
| Increase MF rank from 1 to 5 (ablation) | Forgetting prediction with MF · OLMo-7B-OOD | RMSE: 6.09 -> 5.76 | Helped |
| Increase MF rank from 1 to 8 (ablation) | Forgetting prediction with MF · OLMo-1B | RMSE: 2.85 -> 2.78 | Helped |
| Increase MF rank from 1 to 5 (ablation) | Forgetting prediction with MF · OLMo-1B-OOD | RMSE: 2.86 -> 2.82 | Helped |
| Increase the learning rate from 2e-6 to 1e-6 (ablation) | OLMo-7B fine-tuning and rank-3 association approximation · Dolma forgetting associations | R²: 0.7542 -> 0.7697 | No clear effect |
| Increase the learning rate from 2e-6 to 5e-6 (ablation) | OLMo-7B fine-tuning and rank-3 association approximation · Dolma forgetting associations | R²: 0.7542 -> 0.9582 | Helped |
| Increase batch size from 8 to 32 (ablation) | OLMo-7B fine-tuning and rank-3 association approximation · Dolma forgetting associations | R²: 0.7542 -> 0.8453 | Helped |
Surprising findings
- Gradient inner-product and PPL-threshold selection do not reduce forgetting relative to random replay, suggesting importance definitions differ from forgettability.
- Newer, more capable model families (e.g., OLMo2) show less simple, higher-rank association structure than older families like Pythia.
Limitations
- Analysis limited to supervised fine-tuning and independent (non-sequential) task learning; sequential continual learning not studied.
- Upstream data subsampled (10k to 141k examples) due to compute, and statistics collected in a single run.
- No theoretical guarantee for why example associations are low rank.
Experimental setup
Datasets: Dolma OLMo2-Mix RedPajama The Pile Tulu V2 FLAN Dolly MMLU BBH TruthfulQA OLMo2-SFT-Mix
Benchmarks: Open LLM Leaderboard tasks
Metrics: RMSE F1 R^2 log perplexity increase Token F1 exact match drop
Baselines: SBERT embedding predictor OpenAI text-embedding-3-small predictor Random replay MIR-T PPL threshold selection Gradient inner product selection Ground truth forgetting replay
Builds on: Maximally Interfered Retrieval collaborative filtering first-order Taylor approximation of forgetting
Closest papers by meaning
- What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinement (ICML 2024 · similarity 0.92)
- Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance (NeurIPS 2026 · similarity 0.89)
- Spurious Forgetting in Continual Learning of Language Models (ICLR 2025 · similarity 0.89)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.89)
- Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning (ICLR 2025 · similarity 0.88)
- Learning and Forgetting Unsafe Examples in Large Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.