Research map / Continual learning / Continual fine-tuning of LLMs
Understanding Generalization and Forgetting in In-Context Continual Learning
Guangyu Li, Meng Ding, Lijie Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase in-context examples M per task when earlier tasks are misaligned (ablation) | GPT-2 on linear regression task sequences · synthetic linear regression, tasks 2-5 | normalized MSE | Mixed |
| increase context length M in single-task position (Task 1) (ablation) | GPT-2 on linear regression · synthetic linear regression, Task 1 | prediction error | Improved |
| increase angle between task weight vectors (lower task similarity) (ablation) | GPT-2 on linear regression sequences · synthetic linear regression | normalized MSE | Hurt |
| place a similar task earlier in the prompt (cluster {1,3,5}) (ablation) | GPT-2 on linear regression · synthetic linear regression, 5 tasks | per-task MSE | Improved |
| change task order in the prompt (ablation) | GPT-2 on linear regression · synthetic linear regression, 6 permutations | mean forgetting | Mixed |
| increase M on sparse linear regression and two-layer ReLU tasks (ablation) | GPT-2 · sparse linear regression, Task 5 | normalized MSE: 0.9905 at M = 19 | Hurt |
| prepend SST-2 examples before AG News examples (small M) (ablation) | Qwen2.5-1.5B-Instruct · AG News accuracy | accuracy: 0.736 -> 0.580 (M=1) | Hurt |
| increase M to 19 with prepended SST-2 context (ablation) | Qwen2.5-1.5B-Instruct · AG News accuracy | accuracy: 0.668 -> 0.684 (M=19) | No clear effect |
| append AG News examples after SST-2 examples, query SST-2 at end (ablation) | Qwen2.5-1.5B-Instruct · SST-2 accuracy | accuracy: 0.934 -> 0.472 (M=1) | Hurt |
| increase M from 1 to 19 for Task A retention (ablation) | Qwen2.5-1.5B-Instruct · SST-2 accuracy | accuracy: 0.472 -> 0.536 | Mixed |
Closest papers by meaning
- Differential learning kinetics govern the transition from memorization to generalization during in-context learning (ICLR 2025 · similarity 0.89)
- In-Context Learning Strategies Emerge Rationally (NeurIPS 2025 · similarity 0.89)
- Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability (ICLR 2024 · similarity 0.89)
- Spurious Forgetting in Continual Learning of Language Models (ICLR 2025 · similarity 0.89)
- Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.89)
- Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
