Research map / Continual learning / Continual fine-tuning of LLMs
Convolutional Prompting meets Language Models for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ConvPrompt, a rehearsal-free continual learning method that generates layer-wise prompts by convolving task-shared embeddings with task-specific kernels, and uses GPT-3-generated class attributes to set the number of new prompt generators per task.
Problem
Rehearsal-free class-incremental continual learning with pre-trained vision transformers, where prompt pools share knowledge poorly and use a fixed number of prompts per task regardless of task similarity.
Approach
Prompts are generated per layer and head by single-channel convolution kernels applied to learnable task-shared embedding matrices, with the kernels frozen after each task; a projection network over the layer-wise [CLS] token weights prompt components, and LLM-generated attribute similarity between tasks sets how many new kernels to learn.
Key results
- ConvPrompt reaches 88.87% average accuracy on Split CIFAR-100 (10 tasks) vs 87.00% for CODA-Prompt, with 2.0M trainable parameters vs 4.6M.
- On Split ImageNet-R with 10 tasks, ConvPrompt achieves 77.86% average accuracy and 4.33% forgetting, versus 74.24% for CODA-Prompt.
- On Split CUB-200 (10 tasks), ConvPrompt reaches 80.2% average accuracy vs 74.40% for CODA-Prompt; the paper reports about 3% average gain using roughly 40% of CODA-Prompt's trainable parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add shared embeddings across tasks (ablation) | ViT-B/16 with ConvPrompt modules · ImageNet-R, 10 tasks | average accuracy (AT): 62.1 ± 0.52 -> 67.3 ± 0.58 | Helped |
| Add task-specific prompt generators to shared embeddings (ablation) | ConvPrompt · ImageNet-R, 10 tasks | average accuracy (AT): 67.3 ± 0.58 -> 73.92 ± 0.65 | Helped |
| Use convolutional rather than neural-network prompt generators (ablation) | shared-embedding prompt generation · ImageNet-R, 10 tasks | trainable parameters (%): 14.9/114.9 -> 3.4/103.4 | Helped |
| Add prompt weighting using a projection network (ablation) | ConvPrompt with shared embeddings and convolutional generators · ImageNet-R, 10 tasks | average accuracy (AT): 73.92 ± 0.65 -> 77.96 ± 0.54 | Helped |
| Use a two-layer rather than one-layer projection network (ablation) | ConvPrompt prompt weighting · ImageNet-R, 10 tasks | average accuracy (AT): 76.28 ± 0.23 -> 77.96 ± 0.54 | Helped |
| Add language-driven task similarity to dynamically set prompt-generator count (ablation) | ConvPrompt · ImageNet-R, 10 tasks | average accuracy (AT): 77.96 ± 0.54 -> 77.86 ± 0.25 | No clear effect |
| Use class-attribute similarity instead of class-label similarity (ablation) | ConvPrompt task-similarity module · ImageNet-R, 10 tasks | average accuracy (AT): 76.93 ± 0.45 -> 77.86 ± 0.25 | Helped |
| Use class-attribute similarity instead of image-based similarity (ablation) | ConvPrompt task-similarity module · ImageNet-R, 10 tasks | average accuracy (AT): 77.18 ± 0.42 -> 77.86 ± 0.25 | Helped |
| Increase maximum prompt components per task; reported best is Jmax=5 (ablation) | ConvPrompt · ImageNet-R, 10 tasks | average accuracy (AT) | Mixed |
| Increase prompt length up to 20 (ablation) | ConvPrompt · ImageNet-R, 10 tasks | average accuracy (AT) | Mixed |
| Apply prompts to more transformer layers, peaking at 7 layers (ablation) | ConvPrompt · ImageNet-R, 10 tasks | average accuracy (AT) | Mixed |
| Increase convolution kernel size; reported best is 17 (ablation) | ConvPrompt prompt generators · ImageNet-R, 10 tasks | average accuracy (AT) | Mixed |
Surprising findings
- Adding task-specific convolutional prompt generators improves over the task-shared-only variant by about 6% on ImageNet-R, and language-based task similarity reduces parameters without losing accuracy.
Limitations
- Relies on GPT-3 for class attribute generation, adding external API dependence.
- Shows slightly more forgetting than some baselines on the 10-task Split CUB-200 setup.
Experimental setup
Datasets: ImageNet-R CIFAR-100 CUB-200
Benchmarks: Split CIFAR-100 Split ImageNet-R Split CUB-200
Metrics: average accuracy forgetting
Baselines: Joint-FT Seq-FT ER LwF L2P DualPrompt CODA-Prompt LGCL SLCA
Builds on: L2P DualPrompt CODA-Prompt prefix tuning
Closest papers by meaning
- Evolving Parameterized Prompt Memory for Continual Learning (AAAI 2024 · similarity 0.90)
- Mixture of Experts Meets Prompt-Based Continual Learning (NeurIPS 2024 · similarity 0.90)
- Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning (NeurIPS 2026 · similarity 0.90)
- REP: Resource-Efficient Prompting for Rehearsal-Free Continual Learning (NeurIPS 2025 · similarity 0.90)
- Vector Quantization Prompting for Continual Learning (NeurIPS 2024 · similarity 0.90)
- One Size Fits All for Semantic Shifts: Adaptive Prompt Tuning for Continual Learning (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.