Research map / Continual learning / Continual fine-tuning of LLMs
Memory Efficient Continual Learning with Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use selective Adapter consolidation with a pool of four Adapters instead of single-Adapter distillation (ablation) | ADA · Arxiv | accuracy: 0.772 $\pm$ 0.021 -> 0.898 $\pm$ 0.014 | Helped |
| Use selective Adapter consolidation with a pool of four Adapters instead of single-Adapter distillation (ablation) | ADA · Reuters | accuracy: 0.825 $\pm$ 0.018 -> 0.951 $\pm$ 0.013 | Helped |
| Use selective Adapter consolidation with a pool of four Adapters instead of single-Adapter distillation (ablation) | ADA · Wiki-30K | accuracy: 0.770 $\pm$ 0.019 -> 0.858 $\pm$ 0.011 | Helped |
| Use selective Adapter consolidation with a pool of four Adapters instead of single-Adapter distillation (ablation) | ADA · Mixed | accuracy: 0.729 $\pm$ 0.017 -> 0.812 $\pm$ 0.012 | Helped |
| Select which Adapter to consolidate using TransRate rather than LEEP (ablation) | ADA (K=4) · Arxiv | accuracy: 0.867 $\pm$ 0.017 -> 0.898 $\pm$ 0.014 | Helped |
| Select which Adapter to consolidate using TransRate rather than LEEP (ablation) | ADA (K=4) · Reuters | accuracy: 0.947 $\pm$ 0.015 -> 0.951 $\pm$ 0.013 | Helped |
| Select which Adapter to consolidate using TransRate rather than LEEP (ablation) | ADA (K=4) · Wiki-30K | accuracy: 0.842 $\pm$ 0.013 -> 0.858 $\pm$ 0.011 | Helped |
| Select which Adapter to consolidate using TransRate rather than LEEP (ablation) | ADA (K=4) · Mixed | accuracy: 0.795 $\pm$ 0.012 -> 0.812 $\pm$ 0.012 | Helped |
| Use ADA with TransRate selection rather than AdapterFusion | BERT_base continual classification · Arxiv, Reuters, Wiki-30K, and Mixed | total parameters: 182 M -> 128 M; 326 M -> 128 M; 470 M -> 128 M | Helped |
| Use ADA rather than AdapterFusion for prediction | Reuters · Reuters | inference speedup: ~5 times faster | Helped |
| Use an Adapter-based approach instead of fine-tuning only the classification head | Continual text and image classification · Text datasets and image datasets | accuracy | Helped |
| Fine-tune the full pre-trained model sequentially instead of keeping its representation frozen | B2 full fine-tuning · Continual text and image classification | accuracy | Hurt |
Closest papers by meaning
- Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers (ICML 2025 · similarity 0.88)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.88)
- Learning to Continually Learn with the Bayesian Principle (ICML 2024 · similarity 0.88)
- Online Task-Free Continual Learning via Dynamic Expansionable Memory Distribution (CVPR 2025 · similarity 0.88)
- Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning (CVPR 2025 · similarity 0.87)
- SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.