Research map / Continual learning / Domain and prompt-based
One Size Fits All for Semantic Shifts: Adaptive Prompt Tuning for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaPromptCL, an adaptive prompt-tuning continual learning method that groups semantically similar tasks via assign-and-refine semantic grouping, sharing prompts across mild and abrupt shifts.
Problem
Rehearsal-free prompt-based continual learning uses either one universal prompt or one prompt per task, which fails when task semantic shifts are of mixed (varying) degree, leading to insufficient or redundant prompts.
Approach
Dynamically groups incoming tasks by semantic similarity of warm-up prompts (assignment step), keeps prospective groups with their prompts in reserve, and refines groups by permuting task order within neighboring task sets to reduce the number of groups, retrieving the pre-trained prospective prompts when refinement happens.
Key results
- On VTAB-Rec10 varying shift, AdaPromptCL reaches 54.22% last accuracy versus 52.34% for the best baseline, and it outperforms baselines by up to 21.3% per abstract.
- On uniformly abrupt VTAB-19T, AdaPromptCL achieves 32.39% vs 30.83% for the best baseline, with about 18 prompts.
- Refinement reduces semantic groups from 9.6 to 6.2 on varying benchmarks, with ARI 0.97 and NMI 0.98.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adaptive assign-and-refine prompting instead of S-Prompts | AdaPromptCL · VTAB-Sim25, VTAB-Sim50, VTAB-Sim75, VTAB-Rec2, VTAB-Rec5, and VTAB-Rec10 | last accuracy: 38.29 vs 36.61; 37.92 vs 36.25; 37.59 vs 36.35; 56.40 vs 53.28; 56.86 vs 54.06; 54.22 vs 52.34 | Helped |
| use adaptive assign-and-refine prompting instead of LAE | AdaPromptCL · ImageNet-R | last accuracy: 70.11 vs 69.45 | Hurt |
| use adaptive assign-and-refine prompting instead of VPT | AdaPromptCL · CIFAR100 | last accuracy: 85.34 vs 85.31 | No clear effect |
| use adaptive assign-and-refine prompting instead of S-Prompts | AdaPromptCL · VTAB-19T | last accuracy: 32.39 vs 30.83 | Helped |
| use adaptive assign-and-refine prompting instead of S-Prompts | AdaPromptCL · VTAB-5T | last accuracy: 38.67 vs 38.27 | Helped |
| add semantic refinement (rather than omitting refinement) (ablation) | AdaPromptCL · VTAB-Rec10 | last accuracy: 5.63% average decrease without refinement | Helped |
| add proactive prompt tuning for prospective semantic groups rather than averaging prompts (ablation) | AdaPromptCL · VTAB-Rec10 | last accuracy: 5.63% average decrease for Avg Merge on varying shifts | Helped |
| add semantic refinement rather than using online assignment only (ablation) | AdaPromptCL · VTAB-Rec10 | adjusted Rand index: No Refine: ($\pm$0.031); AdaPromptCL: 0.97 ($\pm$0.005) | Helped |
| add prospective semantic groups and refinement (ablation) | AdaPromptCL · VTAB-Rec10 | GPU running time (minutes): 62.4 to 63.0 | Hurt |
Surprising findings
- Averaging prompts for merged groups (Avg Merge) without proactive tuning degrades accuracy notably on abrupt shifts.
Limitations
- Extra warm-up phase and prospective prompt storage add runtime and memory overhead relative to pure universal methods.
- Sensitivity to threshold R; the paper notes R=0.4 works best for their benchmarks.
- Evaluated only on image classification with ViT backbones and constructed VTAB-based benchmarks.
Experimental setup
Datasets: CIFAR-100 ImageNet-R VTAB-1k (19 tasks)
Benchmarks: CIFAR-100 ImageNet-R VTAB-19T VTAB-5T VTAB-Sim25 VTAB-Sim50 VTAB-Sim75 VTAB-Rec2 VTAB-Rec5 VTAB-Rec10
Metrics: last accuracy forgetting adjusted rand index normalized mutual information GPU runtime
Baselines: L2P VPT DP S-Prompts LAE
Builds on: L2P DualPrompt S-Prompts LAE VPT
Closest papers by meaning
- Convolutional Prompting meets Language Models for Continual Learning (CVPR 2024 · similarity 0.90)
- Adaptive Prompting for Continual Relation Extraction: A Within-Task Variance Perspective (AAAI 2025 · similarity 0.89)
- Consistent Prompting for Rehearsal-Free Continual Learning (CVPR 2024 · similarity 0.88)
- Training Consistent Mixture-of-Experts-Based Prompt Generator for Continual Learning (AAAI 2025 · similarity 0.88)
- Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning (NeurIPS 2026 · similarity 0.88)
- Mixture of Experts Meets Prompt-Based Continual Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.