Research map / Continual learning / Domain and prompt-based
REP: Resource-Efficient Prompting for Rehearsal-Free Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes REP, a resource-efficient wrapper for prompt-based rehearsal-free continual learning on ViTs that uses a ViT-Ti surrogate with random projection for prompt selection, plus adaptive token merging (AToM) and adaptive layer dropping (ALD) during prompt updates to cut training time and memory on edge-constrained devices.
Problem
Prompt-based rehearsal-free continual learning methods with ViT backbones are resource-intensive in both prompt selection (extra forward pass) and prompt update (full forward/backward passes), hindering on-device deployment under tight memory and energy budgets.
Approach
Decouple cost: select prompts with a cheap ViT-Ti surrogate plus fixed random projection, and during prompt updates skip computation non-uniformly—merge non-prompt tokens with a progressive schedule that never merges prompt tokens (AToM), and drop layers with a probability tied to how many tokens were merged in each layer (ALD).
Key results
- REP reduces iteration time by up to 51% and memory by up to 41% across seven prompt-based CL methods and three ViT backbones, with accuracy drops of at most about 1.2% on Split CIFAR-100.
- REP-L2P with ViT-L on Split ImageNet-R keeps accuracy at 75.3% vs 75.6% for L2P while cutting iteration time from 447 ms to 240 ms and memory from 6.5 GB to 4.5 GB.
- Applied to SLCA, REP reduces memory from 9.3 GB to 4.8 GB and iteration time from 1,311 ms to 686 ms, with accuracy rising from 77.2% to 78.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply REP to L2P | L2P with ViT-L · Split ImageNet-R | iteration time: 447(\(\times\)1.9) -> 240 ms | Helped |
| Apply REP to L2P | L2P with ViT-L · Split ImageNet-R | memory: 6.5(\(\times\)1.4) -> 4.5 GB | Helped |
| Apply REP to L2P | L2P with ViT-L · Split ImageNet-R | final average accuracy: 75.6\(\pm\)1.0 -> 75.3\(\pm\)1.5 | No clear effect |
| Apply REP to L2P | L2P with ViT-L · Split PlantDisease | final average accuracy: 75.9\% -> 81.1\% | Helped |
| Add ALD (ablation) | REP-L2P with ViT-L · Split ImageNet-R | iteration time: 270 -> 240 | Helped |
| Add AToM (ablation) | REP-L2P with ViT-L · Split ImageNet-R | memory: 5.5 -> 4.5 | Helped |
| Add AToM and ALD (ablation) | REP-L2P with ViT-L · Split ImageNet-R | final average accuracy: 74.8\(\pm\)0.8 -> 75.3\(\pm\)1.5 | Helped |
| Replace AToM with conventional ToMe (ablation) | REP-L2P with ViT-L · Split ImageNet-R | final average accuracy: 75.3\(\pm\)1.5 -> 70.2\(\pm\)0.7 | Hurt |
| Replace ALD with PLD (ablation) | REP-L2P with ViT-L · Split ImageNet-R | final average accuracy: 75.3\(\pm\)1.5 -> 73.3\(\pm\)0.7 | Hurt |
| Use intermediate ViT-L layer 4 instead of the surrogate for prompt selection (ablation) | REP-L2P with ViT-L · Split ImageNet-R | final average accuracy: 75.3\(\pm\)1.5 -> 74.0\(\pm\)0.8 | Hurt |
| Replace AToM with FIM-based merging (ablation) | REP-L2P with ViT-L · Split ImageNet-R | iteration time: 240 -> 264 | Hurt |
| Replace ALD with 25% random layer dropping (ablation) | REP-L2P with ViT-L · Split ImageNet-R | final average accuracy: 75.8\(\pm\)0.5 -> 70.6\(\pm\)0.4 | Hurt |
Surprising findings
- ToMe-style uniform merging with prompt tokens causes gradient explosion in prompt tokens even with gradient clipping, and the paper reports ALD preserves accuracy where random layer dropping at 25% drops it to about 70%.
- On Split PlantDisease, L2P with ViT-L accuracy rises from 75.9% to 81.1% with REP, despite the efficiency-oriented modifications.
Limitations
- Hyperparameters (tau, n, theta) are tuned by grid search with 5-fold cross-validation per task, limiting plug-and-play use.
- Only evaluated under a fixed task order; no curriculum or task-scheduling analysis.
- Evaluated only on image classification with ImageNet-pretrained ViTs; rehearsal-based variants not explored.
Experimental setup
Datasets: CIFAR-100 ImageNet-R PlantDisease CUB-200
Benchmarks: Split CIFAR-100 Split ImageNet-R Split PlantDisease
Metrics: final average accuracy forgetting iteration time (ms) GPU memory (GB)
Baselines: L2P DualPrompt CODA-Prompt HiDe-Prompt ConvPrompt LAE OVOR SLCA RanPAC Online-LoRA ADAM BudgetCL
Builds on: L2P DualPrompt ToMe progressive layer dropping CODA-Prompt
Closest papers by meaning
- Convolutional Prompting meets Language Models for Continual Learning (CVPR 2024 · similarity 0.90)
- Consistent Prompting for Rehearsal-Free Continual Learning (CVPR 2024 · similarity 0.90)
- Evolving Parameterized Prompt Memory for Continual Learning (AAAI 2024 · similarity 0.90)
- Training Consistent Mixture-of-Experts-Based Prompt Generator for Continual Learning (AAAI 2025 · similarity 0.89)
- Advancing Prompt-Based Methods for Replay-Independent General Continual Learning (ICLR 2025 · similarity 0.88)
- Mixture of Experts Meets Prompt-Based Continual Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.