Research map / Continual learning / Domain and prompt-based
ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuning
Jinyang Zhang, Yue Fang, Hongxin Ding, Weibin Liao, Muyang Ye, Junfeng Zhao, Yasha Wang, Xu Chu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add General-Competence Guided Selective Layer Expansion (Stage 1) (ablation) | ADEPT · Medical CPT, Qwen3-1.7B | MMLU accuracy: 57.31 -> 62.80 | Improved |
| add Adaptive Unit-Wise Decoupled Tuning (Stage 2) (ablation) | ADEPT · Medical CPT, Qwen3-1.7B | MedQA accuracy: 49.23 -> 50.75 | Improved |
| use importance-guided expansion instead of uniform layer insertion (ablation) | ADEPT · Medical CPT, Llama3-8B | MMLU accuracy: 64.19 -> 65.17 | Improved |
| importance-guided expansion plus decoupled tuning versus full-parameter CPT | Qwen3-1.7B-Base math CPT · GSM8K | accuracy: 51.86 -> 70.51 | Improved |
| use ADEPT instead of full-parameter CPT to reduce forgetting | Qwen3-8B-Base math CPT · MMLU | accuracy: 74.90 -> 76.80 | Improved |
| add data replay to full-parameter CPT | PT-Full · Qwen3-1.7B math CPT, MMLU | accuracy: 60.07 -> 60.69 | No clear effect |
| use LoRA CPT instead of full-parameter CPT | PT-LoRA · Qwen3-1.7B medical CPT, MMLU | accuracy: 59.44 -> 57.31 | Hurt |
| update only a small subset of parameters in expanded layers (ADEPT) | training time · Medical CPT, Qwen3-8B | training time: 8 days, 9h -> 3 days, 15h | Improved |
| ADEPT versus Llama Pro on Llama3-8B medical CPT | Llama3-8B-Base · CMB | accuracy: 47.05 -> 61.78 | Improved |
| expand layers before decoupling parameters (ablation) | Qwen3-8B-Base · domain activation separation (KDE) | activation overlap | Improved |
| ADEPT versus full pretraining for token shift | medical CPT · token distribution shift | share of shifted tokens: 5.61% -> 2.18% | Improved |
Closest papers by meaning
- Learning Dynamics in Continual Pre-Training for Large Language Models (ICML 2025 · similarity 0.91)
- Mix-CPT: A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignment (ICLR 2025 · similarity 0.90)
- Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models (NeurIPS 2026 · similarity 0.90)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.89)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.89)
- Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
