Research map / Continual learning / Domain and prompt-based
Evolving Parameterized Prompt Memory for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EvoPrompt, a continual learning method that encodes prompts as an FFN/MLP bottleneck in ViT weights, evolves a reference prompt memory by fusing working prompt memory via optimal-transport alignment and attribution-aware merging, and initializes new classifiers with prototype attention.
Problem
Prompt-based continual learning with discrete key-prompt pools suffers from prompt selection mismatches at test time and limited prompt shareability among similar instances, while pre-trained ViT models still face catastrophic forgetting and classifier incompatibility across tasks.
Approach
Replace discrete prompt-pool selection with a continuous, instance-conditioned prompt generator (MLP bottleneck over the [cls] query) stored in network weights, maintained as a paired reference and working prompt memory that are merged between tasks via OT alignment and attribution-weighted linear fusion (linear mode connectivity), plus prototype-attention-based classifier initialization.
Key results
- On Split CIFAR-100 (5/10/20 steps), EvoPrompt averages 87.19% accuracy with 5.57% forgetting, vs. 86.40% for DualPrompt.
- On Split ImageNet-R, EvoPrompt averages 76.13% accuracy, outperforming CODA-P (72.12%) by about 4 points, with 5.08% forgetting.
- On CORe50 domain-incremental learning, EvoPrompt reaches 95.27% test accuracy, above ESN (91.80%) and the upper bound (91.32%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add FFN prompt memory (MLP bottleneck prompts) (ablation) | ViT-B/16 prompt tuning · Split ImageNet-R 10 steps | Avg Acc: +18.15 over FT-seq (50.28 to 68.43) | Helped |
| add incremental fusion of working into reference prompt memory (ablation) | EvoPrompt · Split ImageNet-R 10 steps | Avg Acc: +4.01 (68.43 to 72.44) | Helped |
| add incremental fusion (ablation) | EvoPrompt · Split ImageNet-R 10 steps | Forgetting: 5.88 to 1.05 | Helped |
| add attribution-aware fusion weighting and OT alignment (ablation) | EvoPrompt · Split ImageNet-R 10 steps | Avg Acc: about +1 (72.44 to 73.16 to 73.51) | Helped |
| add compositional classifier initialization (CCI) (ablation) | EvoPrompt · Split ImageNet-R 10 steps | Avg Acc: +3.32 (73.51 to 76.83) | Helped |
| add compositional classifier initialization (CCI) (ablation) | EvoPrompt · Split ImageNet-R 10 steps | Forgetting: 1.10 to 2.78 | Hurt |
| full EvoPrompt vs DualPrompt baseline | ViT-B/16 prompt CL · Split CIFAR-100 avg over steps | Avg Acc: 86.40 to 87.19 | Helped |
| full EvoPrompt vs CODA-P | ViT-B/16 prompt CL · Split ImageNet-R avg over steps | Avg Acc: 72.12 to 76.13 | Helped |
| full EvoPrompt vs S-liPrompts on CORe50 | ViT-B/16 prompt CL · CORe50 DIL | Test Acc: 89.06 to 95.27 | Helped |
| lighter EvoPrompt-S variant (smaller stored RPM cost) vs full EvoPrompt | EvoPrompt · Split CIFAR-100 avg over steps | Avg Acc: 87.20 vs 87.19 | No clear effect |
| lighter EvoPrompt-S variant vs full EvoPrompt | EvoPrompt · Split ImageNet-R avg over steps | Avg Acc: 75.90 vs 76.13 | No clear effect |
Surprising findings
- The method's CORe50 accuracy exceeds its joint-training upper bound, which the authors attribute to overfitting of the upper bound to the training domains.
Limitations
- Forgetting is higher on Split CIFAR-100 5-step setting (around 10%) than some baselines
- Gains from individual components are modest (about 1% for attribution-aware fusion and alignment)
Experimental setup
Datasets: Split CIFAR-100 Split ImageNet-R CORe50
Benchmarks: Split CIFAR-100 Split ImageNet-R CORe50
Metrics: average accuracy forgetting final test accuracy
Baselines: L2P DualPrompt CODA-P CODA-P-S ESN S-iPrompts S-liPrompts EWC LwF NME-seq
Builds on: L2P DualPrompt VPT CODA-Prompt linear mode connectivity
Closest papers by meaning
- Mixture of Experts Meets Prompt-Based Continual Learning (NeurIPS 2024 · similarity 0.91)
- Convolutional Prompting meets Language Models for Continual Learning (CVPR 2024 · similarity 0.90)
- Training Consistent Mixture-of-Experts-Based Prompt Generator for Continual Learning (AAAI 2025 · similarity 0.90)
- REP: Resource-Efficient Prompting for Rehearsal-Free Continual Learning (NeurIPS 2025 · similarity 0.90)
- Consistent Prompting for Rehearsal-Free Continual Learning (CVPR 2024 · similarity 0.89)
- Visual Prompt Tuning in Null Space for Continual Learning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.