Research map / Continual learning / Continual fine-tuning of LLMs

Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters

CVPR 2024 · Continual learning › Continual fine-tuning of LLMs · approach: Low-rank and model-merging adaptation · 124 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a parameter-efficient continual learning framework that expands a frozen CLIP with incremental mixture-of-experts LoRA adapters and a distribution-based autoencoder selector (DDAS) that routes seen inputs to adapters and unseen inputs to the original CLIP.

Problem

Continual learning of vision-language models suffers from catastrophic forgetting and high training cost of full fine-tuning, while dynamic expansion methods lose zero-shot transfer ability and need task identity at test time.

Approach

Build a dynamic expansion on frozen CLIP using MoE of LoRA adapters with task-specific routers driven by the [CLS] token, an incremental activate-freeze expert strategy, and per-task autoencoders (DDAS) that detect in-distribution vs. out-of-distribution inputs to route between adapters and the original CLIP.

Key results

• On full-shot MTIL, Ours reaches 68.9 Transfer, 76.7 Average, 85.0 Last, versus ZSCL's 68.1/75.4/83.6.
• On 5-shot MTIL, Ours improves Average by 7.0 points (71.4 vs 64.4 for ZSCL) and Last by 4.2 points.
• Training trainable parameters drop from 149.6M (ZSCL) to 59.8M, about 60% reduction, with iteration time reduced about 60%.

Measured effects reported in the paper

Surprising findings

• Adding task-specific routers contributes more to anti-forgetting and zero-shot transfer than simply increasing the number of experts (ablation in Table 6).
• The ablation with a single adapter (no routers) drops Last accuracy to 71.5 and Transfer to 45.0, far below the zero-shot CLIP's Transfer of 69.4 in that row setup.

Limitations

• DDAS relies on a single predefined threshold for all tasks, which may introduce errors as the number of tasks grows.
• Misclassification by DDAS of some Task 9 samples as out-of-distribution occurs, as seen in the t-SNE analysis.
• Improving the zero-shot ability of the original CLIP using learned knowledge is left for future work.

Experimental setup

Datasets: MTIL (Aircraft, Caltech101, CIFAR100, DTD, EuroSAT, Flowers, Food, MNIST, OxfordPet, Cars, SUN397) CIFAR100 TinyImageNet

Benchmarks: MTIL CIFAR100 class-incremental TinyImageNet class-incremental

Metrics: Transfer Average Last accuracy training parameters GPU memory iteration time

Baselines: ZSCL LwF iCaRL LwF-VR WiSE-FT Continual-FT DER DyTox

Builds on: CLIP LoRA ZSCL Mixture of Experts

Closest papers by meaning

• Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models (NeurIPS 2024 · similarity 0.89)
• Continual Vision-Language Retrieval via Dynamic Knowledge Rectification (AAAI 2024 · similarity 0.88)
• CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models (NeurIPS 2024 · similarity 0.88)
• Synthetic Data is an Elegant GIFT for Continual Vision-Language Models (CVPR 2025 · similarity 0.87)
• Enhancing Visual Continual Learning with Language-Guided Supervision (CVPR 2024 · similarity 0.87)
• Stabilizing Zero-Shot Prediction: A Novel Antidote to Forgetting in Continual Vision-Language Tasks (NeurIPS 2024 · similarity 0.87)