Research map / Continual learning / Continual fine-tuning of LLMs
Continual Pre-training of Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DAS instead of Pool for continual domain-adaptive pre-training | RoBERTa · All six end-task datasets (average) | MF1: 77.25 -> 77.93 | Helped |
| Add the contrastive loss (ablation) | DAS · All six end-task datasets (average) | MF1: 76.51 -> 77.93 | Helped |
| Add soft-masking (ablation) | DAS · All six end-task datasets (average) | MF1: 77.19 -> 77.93 | Helped |
| Add initialization that estimates importance for general knowledge (ablation) | DAS · All six end-task datasets (average) | MF1: 76.95 -> 77.93 | Helped |
| Use learned importance scores rather than random scores for soft-masking (ablation) | DAS · All six end-task datasets (average) | MF1: 76.72 -> 77.93 | Helped |
| Contrast full knowledge against previously learned knowledge rather than domain-specific and learned knowledge (ablation) | DAS · All six end-task datasets (average) | MF1: 77.02 -> 77.93 | Helped |
| Use the KL-divergence proxy rather than MLM loss for importance initialization on Wiki (ablation) | DAS · All six end-task datasets (average) | MF1: 76.39 -> 77.75 | Helped |
| Use DAS rather than NCL | RoBERTa continual DAP-training · All six end-task datasets (average) | MF1: 76.36 -> 77.93 | Helped |
Closest papers by meaning
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.89)
- Mix-CPT: A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignment (ICLR 2025 · similarity 0.89)
- Scalable Language Model with Generalized Continual Learning (ICLR 2024 · similarity 0.88)
- CMT: A Memory Compression Method for Continual Knowledge Learning of Large Language Models (AAAI 2025 · similarity 0.88)
- Once Read is Enough: Domain-specific Pretraining-free Language Models with Cluster-guided Sparse Experts for Long-tail Domain Knowledge (NeurIPS 2024 · similarity 0.88)
- Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.