Research map / Continual learning / Continual fine-tuning of LLMs
Towards Continual Knowledge Learning of Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a parameter-expansion method rather than vanilla continued pretraining | T5 · InvariantLAMA | EM: 12.89 -> 20.29 | Helped |
| Use a parameter-expansion method rather than vanilla continued pretraining | T5 · UpdatedLAMA | EM: 10.17 -> 12.77 | Helped |
| Use a parameter-expansion method rather than vanilla continued pretraining | T5 · NewLAMA | EM: 3.77 -> 5.03 | Helped |
| Use a parameter-expansion method rather than vanilla continued pretraining | T5 · NewLAMA-Easy | EM: 17.75 -> 19.56 | Helped |
| Mix replay samples from the original pretraining corpus during continued pretraining | T5-Vanilla · UpdatedLAMA | EM: 10.17 -> 6.49 | Hurt |
| Use Mix-Review instead of vanilla continued pretraining | T5 · CKL benchmark | FUAR: 1.08 -> 1.74 | Hurt |
| Increase K-Adapter count from k=2 to k=3 (ablation) | T5-Kadapters · NewLAMA | EM: 5.03 -> 4.02 | Hurt |
| Increase K-Adapter count from k=2 to k=3 (ablation) | T5-Kadapters · InvariantLAMA | EM: 19.59 -> 19.76 | Helped |
| Train in sequential CKL phases rather than one combined phase | T5-Vanilla · InvariantLAMA | EM: 11.86 -> 9.40 | Hurt |
| Repeat the same training data across more epochs | T5-Vanilla · InvariantLAMA | EM | Hurt |
| Use parameter expansion for CKL | T5 · CKL benchmark | FUAR: 1.08 -> 0.28 | Helped |
Closest papers by meaning
- Train-Attention: Meta-Learning Where to Focus in Continual Knowledge Learning (NeurIPS 2024 · similarity 0.91)
- Spurious Forgetting in Continual Learning of Language Models (ICLR 2025 · similarity 0.91)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.91)
- Scalable Language Model with Generalized Continual Learning (ICLR 2024 · similarity 0.91)
- CMT: A Memory Compression Method for Continual Knowledge Learning of Large Language Models (AAAI 2025 · similarity 0.91)
- Self-Updatable Large Language Models by Integrating Context into Model Parameters (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.