Research map / Continual learning / Online and replay-based
Advancing Prompt-Based Methods for Replay-Independent General Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MISA, which warms up prompt parameters on pretraining data with a forgetting-aware minimization (initial session adaptation) and adds a parameter-free logit mask, to improve prompt-based general continual learning without a replay buffer.
Problem
General continual learning (online, blurry task boundaries) with prompt-based methods on frozen pretrained ViTs suffers from poor initial performance, limited generalizability, and catastrophic forgetting of prompts and output layers, especially without replay.
Approach
Warm up prompts in an initial session on the pretraining data using forgetting-aware minimization (perturbation along the gradient of a pseudo-downstream OOD split, plus an MLP prompt augmentation), and replace learned output masks with a non-parametric batch- or session-level logit mask over present classes.
Key results
- Without replay buffer, MISA reaches 80.98% A_Last on CIFAR-100 (vs MVP 62.59%, +18.39) and 74.84% on Tiny-ImageNet (+22.06 over MVP).
- On ImageNet-R with buffer 0, MISA achieves 50.89 A_AUC and 43.92 A_Last, outperforming MVP by 10.29 and 11.96 points in A_AUC and A_Last respectively.
- Logit mask alone raises CIFAR-100 A_AUC from 67.07 (baseline) to 74.84, and combined with ISA-FAM to 80.55.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MISA instead of MVP without replay | ViT-B/16 GCL · CIFAR-100 | A_Last: 62.59 -> 80.98 | Helped |
| Use MISA instead of MVP without replay | ViT-B/16 GCL · Tiny-ImageNet | A_Last: 52.78 -> 74.84 | Helped |
| Use MISA instead of MVP without replay | ViT-B/16 GCL · ImageNet-R | A_Last: 31.96 -> 43.92 | Helped |
| Add ISA-FAM without logit masking (ablation) | DualPrompt · CIFAR-100 | A_AUC: 67.07 -> 68.97 | Helped |
| Add ISA-FAM without logit masking (ablation) | DualPrompt · ImageNet-R | A_AUC: 40.11 -> 41.09 | Helped |
| Add non-parametric logit masking without ISA-FAM (ablation) | DualPrompt · CIFAR-100 | A_AUC: 67.07 -> 74.84 | Helped |
| Add non-parametric logit masking without ISA-FAM (ablation) | DualPrompt · ImageNet-R | F_Last: 43.27 -> 20.84 | Helped |
| Combine ISA-FAM with non-parametric logit masking (ablation) | DualPrompt · CIFAR-100 | A_AUC: 74.84 -> 80.55 | Helped |
| Combine ISA-FAM with non-parametric logit masking (ablation) | DualPrompt · ImageNet-R | F_Last: 20.84 -> 19.91 | Helped |
| Use naive ISA rather than random prompt initialization (ablation) | DualPrompt with logit mask · CIFAR-100 | A_AUC: 74.84 -> 77.21 | Helped |
| Use SAM instead of naive ISA alone (ablation) | DualPrompt with logit mask · ImageNet-R | A_AUC: 47.71 -> 49.99 | Helped |
| Use FAM instead of SAM in ISA (ablation) | DualPrompt with logit mask · CIFAR-100 | A_AUC: 79.13 -> 80.55 | Helped |
Surprising findings
- Vanilla SAM applied directly to prompt tuning fails to converge well or hinders convergence unless more backbone parameters are unfrozen.
- The batch-level logit mask fails with batch size 1 (A_Last about 1.50 on CIFAR-100 with buffer 0), while session-level mask works better without replay.
Limitations
- Relies on pretraining data (ImageNet-1k) for the initial session; gains shrink or depend on data overlap and domain match (CUB200 ISA gave lower results).
- Batch-level mask fails at batch size 1, so the method is limited for pure one-sample online learning.
- Downstream domain adaptation is not part of the main method and only lightly explored.
Experimental setup
Datasets: ImageNet-1k CIFAR-100 Tiny-ImageNet ImageNet-R NCH CUB200 ImageNet-100 YFCC100M
Benchmarks: CIFAR-100 Tiny-ImageNet ImageNet-R NCH
Metrics: A_AUC A_Last F_Last accuracy
Baselines: Finetuning Linear Probe EWC LwF L2P DualPrompt MVP ER DER++ ER-ACE RM CLIB
Builds on: DualPrompt sharpness-aware minimization Si-Blurry MVP ER-ACE
Closest papers by meaning
- Mixture of Experts Meets Prompt-Based Continual Learning (NeurIPS 2024 · similarity 0.90)
- A Unified and General Framework for Continual Learning (ICLR 2024 · similarity 0.90)
- Stabilizing Zero-Shot Prediction: A Novel Antidote to Forgetting in Continual Vision-Language Tasks (NeurIPS 2024 · similarity 0.89)
- Mitigating Catastrophic Forgetting in Online Continual Learning by Modeling Previous Task Interrelations via Pareto Optimization (ICML 2024 · similarity 0.89)
- GACL: Exemplar-Free Generalized Analytic Continual Learning (NeurIPS 2024 · similarity 0.89)
- Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.