Research map / Continual learning / Continual fine-tuning of LLMs

Overcoming Generic Knowledge Loss with Selective Parameter Update

CVPR 2024 · Continual learning › Continual fine-tuning of LLMs · approach: Continual pretraining of language models · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Selective Parameter Update (SPU), which localizes continual updates of CLIP to the first MLP layer of each transformer block and sparsely updates the ~10% of those parameters with highest gradient-based scores, preserving generic zero-shot knowledge.

Problem

Continually updating foundation vision-language models (CLIP) on new class-incremental tasks causes loss of their generic pre-trained knowledge, which existing continual learning methods largely ignore.

Approach

Localize updates to the first MLP layer of each transformer block (viewed as key-value memories), then select a sparse subset of parameters by a training-free gradient-magnitude score computed before training, and update only those with replay.

Key results

• SPU achieves the best new-task average accuracy increment (21.34%) among compared methods, versus 18.48% for FLYP+ER.
• SPU has a control-set (ImageNet) accuracy drop of only 0.94%, the smallest among methods without external data, comparable to ZSCL which uses CC12M.
• Updating about 10% of the first MLP layer parameters (≈2.7–3.15% of total) gives the best trade-off; a 0.01 selection rate gives C. Drop 1.11 and 0.5 gives Avg. F. 7.76.

Measured effects reported in the paper

Surprising findings

• Random parameter selection achieves nearly the same control-set preservation (C. Drop 0.04) but lower new-task gains, while the gradient-based selection improves over full fine-tuning on new tasks.
• Updating attention layers gives less forgetting but worse new-task accuracy than updating the first MLP layer.

Limitations

• Evaluated only on CLIP ViT-B/16 vision-language classification; generative tasks not explored.
• Relies on a replay buffer (~4% of data) for forgetting mitigation of new tasks.
• Control set accuracy drop (0.94%) is still nonzero and the method is unidirectional across domains.

Experimental setup

Datasets: Birdsnap CUB-200-2011 FGVC-Aircraft Stanford Cars CIFAR100 GTSRB ImageNet

Benchmarks: Birdsnap CUB-200-2011 FGVC-Aircraft Stanford Cars CIFAR100 GTSRB ImageNet (control set)

Metrics: Average accuracy increment (Acc. In.) Average forgetting (Avg. F.) Control set accuracy drop (C. Drop) Accuracy

Baselines: FLYP FLYP+ER MAS LwF PRD LoRA-EWC L2P DualPrompt SLCA ZSCL SparseCL SPG

Builds on: FLYP Experience replay (ER) Causal tracing (ROME-style analysis) Key-value memory view of MLP layers Piggyback

Closest papers by meaning

• Synthetic Data is an Elegant GIFT for Continual Vision-Language Models (CVPR 2025 · similarity 0.88)
• Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.88)
• Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning (ICLR 2024 · similarity 0.88)
• A Practitioner's Guide to Real-World Continual Multimodal Pretraining (NeurIPS 2024 · similarity 0.87)
• LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models (CVPR 2025 · similarity 0.87)
• TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models (ICLR 2025 · similarity 0.87)