Research map / Continual learning / Continual fine-tuning of LLMs
Training Consistent Mixture-of-Experts-Based Prompt Generator for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a mixture-of-experts prompt generator for visual prompt tuning continual learning, trained with null-space orthogonal gradient projections so that prompts for old-task instances stay consistent across tasks.
Problem
Exemplar-free class-incremental learning with prompt-based ViT models suffers from forgetting because prompt generators drift; existing methods rely on freezing prompts, slow updates, or task discrimination without theoretical guarantees of consistent instance-aware prompts.
Approach
Derives sufficient orthogonality conditions under which an MoE prompt generator (router plus experts) yields identical prompts for old-task inputs after updates in a new task, and enforces them by projecting router and expert gradients onto null spaces of old input features and old gating vectors, with a relaxed weight for expert projection.
Key results
- VPT-CPG reaches 90.63% accuracy on 10-split CIFAR-100 with 3.98% forgetting, versus 84.68% and 15.36% for the VPT-baseline.
- VPT-CPG surpasses the strongest compared method by up to 1.76% and by 0.91% on average across four benchmarks, reaching 83.21% on 10-split DomainNet.
- In long-term settings (50/100 tasks), VPT-CPG improves accuracy by 12%-35% over the VPT-baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the MoE prompt generator instead of using one expert without routing (ablation) | VPT-baseline · 10-split CIFAR-100 | accuracy: 84.68±0.23 -> 86.08±0.52 | Helped |
| Add router projection Hw without expert projection Hp (ablation) | MoE prompt generator · 10-split CIFAR-100 | accuracy: 86.08±0.52 -> 85.72±0.41 | Hurt |
| Add expert projection Hp without router projection Hw (ablation) | MoE prompt generator · 10-split CIFAR-100 | accuracy: 86.08±0.52 -> 88.17±0.49 | Helped |
| Use both router projection Hw and expert projection Hp (ablation) | MoE prompt generator · 10-split CIFAR-100 | accuracy: 84.68±0.23 -> 90.63±0.44 | Helped |
| Use both router projection Hw and expert projection Hp (ablation) | MoE prompt generator · 10-split CIFAR-100 | Forgetting: 15.36±0.34 -> 3.98±0.65 | Helped |
| Use expert projection weight η=0.99 rather than η=1 (ablation) | VPT-CPG · 10-split CIFAR-100 | accuracy | Helped |
Surprising findings
- Using only the router projection (Hw) without the expert projection can increase forgetting relative to the MoE alone on some benchmarks, so the two projections must be used jointly.
Limitations
- Consistency guarantee relies on sufficient (not necessary) conditions and on the null-space threshold hyperparameters.
- Expert orthogonality is relaxed with weight eta (0.99), trading some stability for plasticity.
Experimental setup
Datasets: CIFAR-100 ImageNet-R DomainNet
Benchmarks: 10-split CIFAR-100 20-split CIFAR-100 10-split ImageNet-R 10-split DomainNet 50-task and 100-task CIL settings
Metrics: average accuracy forgetting
Baselines: L2P DualPrompt CODA-Prompt LAE LGCL C-LN EvoPrompt ESN HSICBO PGP OVOR-Deep ConvPrompt
Builds on: visual prompt tuning NSCL L2P Sparsely-gated mixture of experts
Closest papers by meaning
- Consistent Prompting for Rehearsal-Free Continual Learning (CVPR 2024 · similarity 0.92)
- Mixture of Experts Meets Prompt-Based Continual Learning (NeurIPS 2024 · similarity 0.91)
- Visual Prompt Tuning in Null Space for Continual Learning (NeurIPS 2024 · similarity 0.90)
- Evolving Parameterized Prompt Memory for Continual Learning (AAAI 2024 · similarity 0.90)
- Convolutional Prompting meets Language Models for Continual Learning (CVPR 2024 · similarity 0.89)
- REP: Resource-Efficient Prompting for Rehearsal-Free Continual Learning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.