Research map / Continual learning / Domain and prompt-based
Vector Quantization Prompting for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VQ-Prompt, a prompt-based class-incremental continual learning method that quantizes a soft-selected continuous prompt to its nearest discrete element in a prompt pool, trained end-to-end with straight-through gradients and VQ/commitment losses plus classifier bias mitigation via Gaussian representation statistics.
Problem
Prompt-based continual learning relies on non-differentiable prompt identity selection, so prompt selection cannot be optimized with task loss; prior differentiable alternatives produce continuous prompts that lack abstraction for task knowledge.
Approach
Form a continuous prompt as a softmax-weighted sum over a prompt pool, then replace it with its nearest prompt-pool element (vector quantization), training end-to-end via straight-through gradient estimation plus VQ and commitment losses, so discrete prompts are optimized with task loss.
Key results
- On 10-task ImageNet-R (ImageNet-1K pretrained ViT-B), VQ-Prompt reaches 78.71 FAA and 83.24 CAA, above EvoPrompt's 76.83 and 82.09.
- On 20-task ImageNet-R, VQ-Prompt achieves 78.10 FAA versus 74.41 for EvoPrompt and 76.46 for HiDe-Prompt.
- Replacing discrete quantization with a low-temperature soft prompt (Soft-Prompt) yields at best 77.15 FAA on 10-task ImageNet-R, below VQ-Prompt-S (78.05) and VQ-Prompt (78.83).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace continuous soft prompt selection with VQ discrete prompt selection (ablation) | VQ-Prompt on ImageNet-R 10-task · ImageNet-R 10-task | FAA: 77.15 (Soft-Prompt, $\tau\!=\!1.0$) -> 78.05 (VQ-Prompt-S) | Helped |
| add representation statistics for classifier bias mitigation (ablation) | VQ-Prompt-S on ImageNet-R · ImageNet-R 10-task | FAA: 78.00 $\pm$ 0.39 -> 78.71 $\pm$ 0.22 | Helped |
| add representation statistics for classifier bias mitigation (ablation) | VQ-Prompt-S on ImageNet-R · ImageNet-R 5-task | FAA: 78.52 $\pm$ 0.34 -> 79.23 $\pm$ 0.29 | Helped |
| add representation statistics for classifier bias mitigation (ablation) | VQ-Prompt-S on ImageNet-R · ImageNet-R 20-task | FAA: 76.19 $\pm$ 0.26 -> 78.10 $\pm$ 0.22 | Helped |
| add representation statistics for classifier bias mitigation (ablation) | L2P++ on ImageNet-R · ImageNet-R 5-task | FAA: 70.83 $\pm$ 0.58 -> 74.11 $\pm$ 0.08 | Helped |
| add representation statistics for classifier bias mitigation (ablation) | L2P++ on ImageNet-R · ImageNet-R 10-task | FAA: 69.29 $\pm$ 0.73 -> 72.93 $\pm$ 0.27 | Helped |
| add representation statistics for classifier bias mitigation (ablation) | L2P++ on ImageNet-R · ImageNet-R 20-task | FAA: 65.89 $\pm$ 1.30 -> 70.99 $\pm$ 0.26 | Helped |
| reduce prompt length $L_p$ to excessively small values (ablation) | VQ-Prompt · ImageNet-R 10-task | FAA | Hurt |
| increase prompt pool size $N$ while prompt length is small (ablation) | VQ-Prompt · ImageNet-R 10-task | FAA | Helped |
| increase prompt length $L_p$ beyond a certain threshold (ablation) | VQ-Prompt · ImageNet-R 10-task | FAA | Mixed |
| vary VQ and commitment loss weights (ablation) | VQ-Prompt · ImageNet-R 10-task | FAA | No clear effect |
Surprising findings
- Lowering the softmax temperature to approximate discrete selection did not help; the best Soft-Prompt result was at tau=1.0.
Limitations
- Depends on large pre-trained models, inheriting pre-training data limits and high compute cost.
- No constraints on prompt selection, so some prompts are overused across tasks.
Experimental setup
Datasets: ImageNet-R CIFAR-100 CUB-200-2011 ImageNet-1K ImageNet-21K ImageNet-A VTAB
Benchmarks: ImageNet-R Split CIFAR-100 Split CUB-200 ImageNet-A VTAB
Metrics: FAA CAA
Baselines: FT FT++ L2P++ Deep L2P++ DualPrompt CODA-Prompt HiDe-Prompt EvoPrompt LwF BiC SLCA
Builds on: CODA-Prompt L2P DualPrompt VQ-VAE prefix tuning
Closest papers by meaning
- Convolutional Prompting meets Language Models for Continual Learning (CVPR 2024 · similarity 0.90)
- Consistent Prompting for Rehearsal-Free Continual Learning (CVPR 2024 · similarity 0.89)
- Evolving Parameterized Prompt Memory for Continual Learning (AAAI 2024 · similarity 0.88)
- Training Consistent Mixture-of-Experts-Based Prompt Generator for Continual Learning (AAAI 2025 · similarity 0.88)
- Mixture of Experts Meets Prompt-Based Continual Learning (NeurIPS 2024 · similarity 0.88)
- Advancing Prompt-Based Methods for Replay-Independent General Continual Learning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.