Research map / Continual learning / Continual fine-tuning of LLMs
Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEMA, a rehearsal-free continual learning method that self-expands a frozen ViT-B/16 with modular adapters (functional adapter plus autoencoder representation descriptor) triggered by z-score novelty detection, and mixes adapters with an expandable router.
Problem
Continual learning with pre-trained models must balance stability and plasticity; fixed adapter/prompt pools limit adaptation, while task-specific expansion grows model size linearly and impairs knowledge reuse.
Approach
Modular adapters pair a functional adapter with an autoencoder representation descriptor that detects distribution shift per layer; a new adapter is added only when all existing descriptors flag the input as novel (z-score threshold), and an expandable softmax router mixes adapter outputs.
Key results
- On ImageNet-A, SEMA reaches AN 53.32 and average incremental accuracy 64.53, versus 52.60-ish for InfLoRA's 46.21 AN baseline (InfLoRA AN 46.21 and ADAM AN 49.24).
- On VTAB, SEMA achieves AN 89.64 vs 87.63 for InfLoRA and 83.61 for ADAM.
- On 20-task ImageNet-R, SEMA reaches AN 74.53 vs 72.60 for InfLoRA (AN) and 61.35 for ADAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SEMA instead of the strongest reported baseline | ViT-B/16-IN1K · CIFAR-100 | average accuracy at last step: 86.11 -> 86.98 | Helped |
| use SEMA instead of the strongest reported baseline | ViT-B/16-IN1K · 5-Task ImageNet-R | average accuracy at last step: 76.98 -> 79.78 | Helped |
| use SEMA instead of the strongest reported baseline | ViT-B/16-IN1K · 10-Task ImageNet-R | average accuracy at last step: 75.32 -> 78.00 | Helped |
| use SEMA instead of the strongest reported baseline | ViT-B/16-IN1K · 20-Task ImageNet-R | average accuracy at last step: 72.60 -> 74.53 | Helped |
| use SEMA instead of the strongest reported baseline | ViT-B/16-IN1K · ImageNet-A | average accuracy at last step: 50.23 -> 53.32 | Helped |
| use SEMA instead of the strongest reported baseline | ViT-B/16-IN1K · VTAB | average accuracy at last step: 87.63 -> 89.64 | Helped |
| add self-expansion rather than keeping only first-session adaptation (ablation) | SEMA · ImageNet-A | average accuracy at last step: 49.90 -> 53.32 | Helped |
| add self-expansion rather than keeping only first-session adaptation (ablation) | SEMA · VTAB | average accuracy at last step: 83.66 -> 89.64 | Helped |
| use a learned weighting router instead of average adapter weights (ablation) | SEMA · ImageNet-A | average accuracy at last step: 44.31 -> 53.32 | Helped |
| use a learned weighting router instead of random adapter weights (ablation) | SEMA · ImageNet-A | average accuracy at last step: 49.77 -> 53.32 | Helped |
| use a learned soft mixture rather than selecting only the highest-weight adapter (ablation) | SEMA · ImageNet-A | average accuracy at last step: 50.56 -> 53.32 | Helped |
| replace the learned router with random adapter weights (ablation) | SEMA · VTAB | average accuracy at last step: 89.64 -> 85.17 | Hurt |
Surprising findings
- Restricting expansion to a single layer is less effective than multi-layer expansion, and expanding earlier layers adds many adapters without much accuracy gain.
Limitations
- Expansion is limited to at most one adapter per layer per task, so it is not fully online.
- Expansion relies on the quality of representation descriptor distribution shift detection, which the authors suggest could be improved.
- Experiments are restricted to class-incremental image classification with ViT-B/16 ImageNet-1K weights.
Experimental setup
Datasets: CIFAR-100 ImageNet-R ImageNet-A VTAB
Benchmarks: CIFAR-100 ImageNet-R 5-task ImageNet-R 10-task ImageNet-R 20-task ImageNet-A VTAB
Metrics: average accuracy AN average incremental accuracy
Baselines: FT Adapter L2P DualPrompt CODA-P SimpleCIL ADAM InfLoRA
Builds on: ADAM AdaptFormer L2P DualPrompt CODA-Prompt
Closest papers by meaning
- Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.89)
- CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning (CVPR 2025 · similarity 0.88)
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.88)
- MOS: Model Surgery for Pre-Trained Model-Based Class-Incremental Learning (AAAI 2025 · similarity 0.88)
- Online Task-Free Continual Learning via Dynamic Expansionable Memory Distribution (CVPR 2025 · similarity 0.88)
- BECAME: Bayesian Continual Learning with Adaptive Model Merging (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.