Research map / Continual learning / Continual fine-tuning of LLMs

Stabilizing Zero-Shot Prediction: A Novel Antidote to Forgetting in Continual Vision-Language Tasks

NeurIPS 2024 · Continual learning › Continual fine-tuning of LLMs · approach: Low-rank and model-merging adaptation · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ZAF, a replay-free continual learning method for BLIP vision-language models that uses a zero-shot stability regularizer on unpaired wild data plus an EMA-LoRA architecture, and shows it beats baselines on SVLC continual benchmarks.

Problem

Pre-trained vision-language models forget previously learned reasoning skills when continually adapted to sequential tasks, especially without replay or task identifiers, and existing methods trade learning against forgetting.

Approach

Empirically and theoretically links zero-shot prediction stability to anti-forgetting, then adds a zero-shot stability loss on synthetic unpaired wild image-text data, using an EMA of LoRA adapters to provide the previous model's supervision without storing past data.

Key results

• ZAF on BLIP w/ NLVR reaches FAA 89.67 on 7 Task VG+VAW, leading the strongest baseline ConStruct-VL by 3.70 points, with 4.82 and 4.38 leads on the other two benchmarks.
• ZAF trains 29.67x, 17.66x, and 10.98x faster than ConStruct-VL on the three benchmarks.
• Adding the zero-shot antidote as a plug-in improves existing methods, with average FFM reduction of 14.52% on 7 Task VG.

Measured effects reported in the paper

Surprising findings

• Zero-shot stability on unseen future tasks tracks anti-forgetting ability across CL methods.
• Ungrammatical wild texts degrade performance, as the model tends to ignore the image.

Limitations

• The antidote presumes a feasible underlying CL framework and is designed for a sequence of downstream tasks, limiting applicability to the original pre-training contexts.
• Evaluated only on BLIP-family models and SVLC reasoning benchmarks, with a single hyperparameter alpha selected at 0.85.

Experimental setup

Datasets: Visual Genome VAW SVLC benchmark (ConStruct-VL splits) Wild dataset (generated with ChatGPT)

Benchmarks: 7 Task VG+VAW 7 Task VG 5 Task VAW 4 Task VAW

Metrics: FAA CAA FFM training time

Baselines: Continual-FT LoRA Layered-LoRA LwF ZSCL MoE-Adapters ConStruct-VL L2 EWC L2P Dual Prompt

Builds on: LoRA Layered-LoRA ConStruct-VL ZSCL MoE-Adapters LwF

Closest papers by meaning

• Re-evaluating Continual Learning with Few-Shot Adaptation (NeurIPS 2026 · similarity 0.90)
• ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think (ICML 2025 · similarity 0.90)
• Advancing Prompt-Based Methods for Replay-Independent General Continual Learning (ICLR 2025 · similarity 0.89)
• Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning (NeurIPS 2026 · similarity 0.89)
• STAR: Stability-Inducing Weight Perturbation for Continual Learning (ICLR 2025 · similarity 0.89)
• PaLoRA: Paced Low-Rank Adaptation for Continual Learning (NeurIPS 2026 · similarity 0.89)