Research map / Continual learning / Continual fine-tuning of LLMs
Mixture of Experts Meets Prompt-Based Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that prefix tuning in ViT attention is a mixture of experts with linear gating and proposes NoRGa, a non-linear residual gate on prefix scores, which improves HiDe-Prompt on class-incremental benchmarks.
Problem
Prompt-based continual learning methods work well empirically but lack theoretical explanation of why prompts help; prefix tuning's linear gating in the MoE view has poor sample efficiency for parameter estimation.
Approach
Views ViT self-attention heads as mixture-of-experts with quadratic gating and prefix tuning as adding new linear-gated experts; proposes NoRGa, replacing the linear prefix gate with a non-linear residual gate (score + alpha*sigma(tau*score)), giving faster parameter estimation rates while keeping the parameter count.
Key results
- NoRGa reaches 94.48% FA on Split CIFAR-100 with Sup-21K ViT, versus 92.61% for HiDe-Prompt.
- On Split CUB-200 with Sup-21K, NoRGa achieves 90.90% FA versus 86.56% for HiDe-Prompt.
- On Split ImageNet-R with Sup-21K, NoRGa improves CA to 79.52% from 76.60% for HiDe-Prompt.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace HiDe-Prompt's linear prompt gating with NoRGa nonlinear residual gates | HiDe-Prompt · Split CIFAR-100 across Sup-21K, iBOT-21K, iBOT-1K, DINO-1K, and MoCo-1K | FA: 92.61 -> 94.48; 93.02 -> 94.76; 93.48 -> 94.01; 92.51 -> 93.43; 91.57 -> 93.52 | Helped |
| replace HiDe-Prompt's linear prompt gating with NoRGa nonlinear residual gates | HiDe-Prompt · Split ImageNet-R across Sup-21K, iBOT-21K, iBOT-1K, DINO-1K, and MoCo-1K | FA: 75.06 -> 75.40; 70.83 -> 73.06; 71.33 -> 72.77; 68.11 -> 71.77; 63.77 -> 64.52 | Helped |
| replace HiDe-Prompt's linear prompt gating with NoRGa nonlinear residual gates | HiDe-Prompt · Split CUB-200 (Sup-21K and iBOT-21K) | FA: 86.56 -> 90.90; 78.23 -> 80.69 | Helped |
| replace HiDe-Prompt's linear prompt gating with NoRGa nonlinear residual gates | HiDe-Prompt · 5-Datasets (Sup-21K and iBOT-21K) | FA: 93.83 -> 94.16; 94.88 -> 94.92 | Helped |
| use tanh activation in NoRGa (ablation) | NoRGa · Split CIFAR-100 (Sup-21K and iBOT-21K) | FA: 92.61 -> 94.36; 93.02 -> 94.76 | Helped |
| use tanh activation in NoRGa (ablation) | NoRGa · Split CUB-200 (Sup-21K and iBOT-21K) | FA: 86.56 -> 90.87; 78.23 -> 80.69 | Helped |
| use sigmoid activation in NoRGa (ablation) | NoRGa · Split CIFAR-100 (Sup-21K and iBOT-21K) | FA: 92.61 -> 94.48; 93.02 -> 94.69 | Helped |
| use sigmoid activation in NoRGa (ablation) | NoRGa · Split CUB-200 (Sup-21K and iBOT-21K) | FA: 86.56 -> 90.90; 78.23 -> 80.18 | Helped |
| use GELU activation in NoRGa (ablation) | NoRGa · Split CIFAR-100 (Sup-21K and iBOT-21K) | FA: 92.61 -> 94.05; 93.02 -> 94.63 | Helped |
| use GELU activation in NoRGa (ablation) | NoRGa · Split CUB-200 (Sup-21K and iBOT-21K) | FA: 86.56 -> 90.74; 78.23 -> 80.54 | Helped |
Limitations
- Choice of activation function (tanh, sigmoid, GELU) requires tuning.
- Expert network kept simple; more intricate expert architectures are not explored.
Experimental setup
Datasets: Split CIFAR-100 Split ImageNet-R Split CUB-200 5-Datasets
Benchmarks: Split CIFAR-100 Split ImageNet-R Split CUB-200 5-Datasets
Metrics: final average accuracy cumulative average accuracy average forgetting
Baselines: L2P DualPrompt CODA-Prompt S-Prompt HiDe-Prompt ADAM RanPAC
Builds on: HiDe-Prompt prefix tuning DualPrompt L2P
Closest papers by meaning
- Evolving Parameterized Prompt Memory for Continual Learning (AAAI 2024 · similarity 0.91)
- Training Consistent Mixture-of-Experts-Based Prompt Generator for Continual Learning (AAAI 2025 · similarity 0.91)
- Advancing Prompt-Based Methods for Replay-Independent General Continual Learning (ICLR 2025 · similarity 0.90)
- Convolutional Prompting meets Language Models for Continual Learning (CVPR 2024 · similarity 0.90)
- Visual Prompt Tuning in Null Space for Continual Learning (NeurIPS 2024 · similarity 0.90)
- Consistent Prompting for Rehearsal-Free Continual Learning (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.