Research map / Continual learning / Continual fine-tuning of LLMs
SLCA++: Unleash the Power of Sequential Fine-Tuning for Continual Learning with Pre-training
Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, Yunchao Wei
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Slow Learner (reduced representation learning rate) (ablation) | Seq FT · Split CIFAR-100 (IN21K-Sup) | Last-Acc: 41.77 -> 88.86 | Improved |
| add Slow Learner (ablation) | Seq FT · Split ImageNet-R (IN21K-Sup) | Last-Acc: 26.95 -> 71.80 | Improved |
| add Slow Learner (ablation) | Seq FT · Split CUB-200 (IN21K-Sup) | Last-Acc: 40.02 -> 68.07 | Improved |
| add Slow Learner (ablation) | Seq FT · Split Cars-196 (IN21K-Sup) | Last-Acc: 27.57 -> 49.74 | Improved |
| update representation layer with SL instead of fixing it (ablation) | Seq FT · Split CIFAR-100 (IN21K-Sup) | Last-Acc: 63.75 -> 88.86 | Improved |
| add classifier alignment and logit normalization to SL (ablation) | Seq FT w/ SL · Split CUB-200 (IN21K-Sup) | Last-Acc: 68.07 -> 84.71 | Improved |
| add classifier alignment and logit normalization to SL (ablation) | Seq FT w/ SL · Split Cars-196 (IN21K-Sup) | Last-Acc: 49.74 -> 67.73 | Improved |
| add logit normalization to CA (ablation) | SL+CA · Split Cars-196 (IN21K-Sup) | Last-Acc: 67.90 -> 67.73 | Mixed |
| add symmetric cross-entropy (ablation) | SL+CA+LN · Split Cars-196 (IN21K-Sup) | Last-Acc: 67.73 -> 72.20 | Improved |
| add symmetric cross-entropy (ablation) | SL+CA+LN · Split CIFAR-100 (IN1K-Self) | Last-Acc: 85.27 -> 85.12 | Hurt |
| align classifier during representation learning rather than post-hoc (ablation) | CA · Split CIFAR-100 | Last-Acc: -27.89% | Hurt |
| initialize LoRA and classifier (ablation) | Hybrid SL · Split Cars-196 (IN21K-Sup) | Last-Acc: 65.37 -> 73.97 | Improved |
Closest papers by meaning
- SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models (NeurIPS 2024 · similarity 0.89)
- LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning (ICLR 2025 · similarity 0.88)
- Learning to Continually Learn with the Bayesian Principle (ICML 2024 · similarity 0.87)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.87)
- RanDumb: Random Representations Outperform Online Continually Learned Representations (NeurIPS 2024 · similarity 0.87)
- LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learning (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
