Research map / Continual learning / Continual fine-tuning of LLMs
SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAFE, a slow-and-fast parameter-efficient tuning framework for replay-free class-incremental learning with pre-trained ViTs, combining a PTM-correlation transfer loss for a frozen slow learner with a cross-classification and feature-alignment-guided fast learner, fused by entropy-based aggregation.
Problem
Continual learning with pre-trained models: PET applied in the first session loses PTM general knowledge, and freezing parameters in later sessions limits plasticity for novel classes, all under replay-free class-incremental learning.
Approach
A slow PET learner is calibrated in session 1 with a cross-correlation transfer loss against the frozen PTM and then frozen; a fast PET learner is trained in later sessions, guided by the slow learner via feature alignment and cross-classification losses; predictions aggregate both learners' logits weighted by entropy-based confidence.
Key results
- SAFE reaches 84.6% average final accuracy over six class-incremental datasets, versus 82.5% for RanPAC (best baseline in the table).
- On ImageNet-A, SAFE surpasses the second-best result by more than 4% (66.6 vs 62.2 for SLCA/RanPAC-level baselines in the table).
- On DomainNet (domain-incremental), SAFE reaches 67.82% final accuracy, 1.2% above the next best (RanPAC 66.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SAFE instead of RanPAC | ViT-B/16-IN1K continual learning · ImageNet-A final session | accuracy: 61.8 -> 66.6 | Helped |
| Use SAFE instead of RanPAC | ViT-B/16-IN1K continual learning · Six class-incremental datasets average | average accuracy: 82.5 -> 84.6 | Helped |
| Add the slow learner to the baseline (ablation) | SAFE on ImageNet-A · ImageNet-A | final accuracy: 62.21 -> 65.44 | Helped |
| Add the fast learner to the baseline (ablation) | SAFE on ImageNet-A · ImageNet-A | final accuracy: 62.21 -> 66.49 | Helped |
| Combine slow and fast learners (ablation) | SAFE on ImageNet-A · ImageNet-A | final accuracy: 66.49 -> 66.56 | Helped |
| Use entropy-based aggregation instead of feature concatenation (ablation) | Slow and fast learners · ImageNet-A | final accuracy: 65.59 -> 66.56 | Helped |
| Use entropy-based aggregation instead of logits addition or logits max (ablation) | Slow and fast learners · ImageNet-A | final accuracy: 65.90 -> 66.56; 66.03 -> 66.56 | Helped |
| Transfer PTM knowledge using the slow learner loss instead of feature, logits, or second-order statistics alignment (ablation) | Slow learner · ImageNet-A | final accuracy: 62.81 -> 65.44; 64.06 -> 65.44; 63.20 -> 65.44 | Helped |
| Add cross-classification loss to fast learner fine-tuning (ablation) | Fast learner · ImageNet-A | final accuracy: 62.21 -> 65.31 | Helped |
| Add feature alignment loss to fast learner fine-tuning (ablation) | Fast learner · ImageNet-A | final accuracy: 62.21 -> 66.07 | Helped |
| Add cross-classification and feature alignment losses instead of direct fine-tuning (ablation) | Fast learner · ImageNet-A | final accuracy: 8.16 -> 66.49 | Helped |
| Increase λ_cos from 0 to 50 (ablation) | Fast learner · ImageNet-A | final accuracy: 18.56 -> 66.49 | Helped |
Surprising findings
- Finetuning the fast learner without the cross-classification or feature-alignment losses collapses final accuracy on IN-A to 8.16%.
Limitations
- Relies on a strong feature extractor, making it less suitable when training from scratch or on small tasks.
- Introduces several hyper-parameters (lambda_diag, lambda_rdn, lambda_cos, gamma) whose defaults may not transfer to datasets with different statistics.
Experimental setup
Datasets: CIFAR100 ImageNet-R ImageNet-A CUB200 Omnibenchmark VTAB DomainNet
Benchmarks: CIFAR100 ImageNet-R ImageNet-A CUB200 Omnibenchmark VTAB DomainNet
Metrics: final accuracy average accuracy
Baselines: L2P DualPrompt CODAPrompt ADaM RanPAC SSIAT SLCA EASE S-iPrompts
Builds on: RanPAC complementary learning systems Adapter SSF VPT
Closest papers by meaning
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.90)
- Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.90)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.90)
- STAR: Stability-Inducing Weight Perturbation for Continual Learning (ICLR 2025 · similarity 0.89)
- InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning (CVPR 2024 · similarity 0.89)
- Large Continual Instruction Assistant (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.