Research map / Continual learning / Continual fine-tuning of LLMs
CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLAP4CLIP, a variational probabilistic finetuning framework for CLIP in class-incremental learning that models visual-guided text features with task-specific probabilistic adapters, plus language-aware regularization and initialization to reduce forgetting.
Problem
Deterministic finetuning of CLIP in class-incremental learning overlooks cross-modal uncertainty, leading to cross-modal feature deviation, forgetting, and poor calibration, and existing probabilistic finetuning methods are incompatible with prompt-based methods.
Approach
Place a variational posterior over visual-guided text features (a shared visual-guided attention module feeding task-specific probabilistic adapters as a mixture of experts), with forgetting mitigated by regularizing past-task distributions via hand-crafted prompt features and initializing adapters from text feature statistics.
Key results
- On CIFAR100 (ViT-B/16), Ours reaches 86.13 Avg and 78.21 Last, versus 84.84/76.55 for PROOF.
- On ImageNet-R, AttriCLIP + Ours reaches 86.35 Avg, outperforming PROOF by about 1.46 points.
- On the CDCL setting with ViT-L/14, AttriCLIP + Ours achieves 84.56 last-task accuracy on CIFAR100+ImageNet100, above PROOF's 82.59.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the VGA module to the probabilistic encoder (ablation) | CLAP · CIFAR100 | Last accuracy: 11.49 -> 73.41 | Helped |
| Add memory consolidation training (ablation) | CLAP with VGA · CIFAR100 | Last accuracy: 73.41 -> 74.99 | Helped |
| Add task-specific encoders (ablation) | CLAP with consolidation training · CIFAR100 | Last accuracy: 74.99 -> 77.2 | Helped |
| Add language-aware weight initialization (ablation) | CLAP with task-specific encoders · CIFAR100 | Last accuracy: 77.2 -> 77.98 | Helped |
| Add language-aware distribution regularization (ablation) | CLAP with weight initialization · CIFAR100 | Last accuracy: 77.98 -> 78.21 | Helped |
| Use variational inference instead of deterministic inference (ablation) | CLAP · CIFAR100 | Last accuracy: 76.8 -> 78.21 | Helped |
| Add language-aware weight initialization (ablation) | AttriCLIP + Ours, replay-free CL · CIFAR100 | Last accuracy: 60.35 -> 69.39 | Helped |
| Add language-aware distribution regularization (ablation) | AttriCLIP + Ours, replay-free CL · CIFAR100 | Last accuracy: 53.01 -> 69.39 | Helped |
| Add language-aware distribution regularization (ablation) | AttriCLIP + Ours, computationally budgeted CL · CIFAR100 | Last accuracy: 52.6 -> 55.89 | Helped |
| Add language-aware weight initialization (ablation) | AttriCLIP + Ours, computationally budgeted CL · CIFAR100 | Last accuracy: 54.95 -> 55.89 | Helped |
| Use language-aware rather than static prior (ablation) | CLAP · CIFAR100 | ECE: 0.204 -> 0.214 | Hurt |
| Use language-aware rather than static prior (ablation) | CLAP · CIFAR100 | Last accuracy: 78.21 -> 78.38 | Helped |
Surprising findings
- Variance of the softmax outputs as an exemplar selection criterion outperforms random selection and is second only to iCaRL in Last accuracy.
- Reported in the paper's BwT table, CLAP variants are the only methods with positive BwT on the VTAB cross-dataset setting.
Limitations
- Inference time is higher than other finetuning methods due to the VGA module and the M Monte Carlo samples.
- Performance is sensitive to the number of MC samples and VGA/encoder depth; deeper task encoders reduce accuracy.
- Ours w/o VI and some variants show weaker results on some datasets; some variants (e.g., AttriCLIP + Ours on CIFAR100 Last) underperform their baselines in places.
Experimental setup
Datasets: CIFAR100 ImageNet100 ImageNet-R CUB200 VTAB CDCL (ImageNet100 + CIFAR100)
Benchmarks: CIFAR100 ImageNet100 ImageNet-R CUB200 VTAB CDCL
Metrics: Avg accuracy Last accuracy BwT ECE forward transfer AUROC AUPR FPR95
Baselines: Continual-CLIP CoOp CLIP-Adapter AttriCLIP MaPLe PROOF DualPrompt L2P CODA-P iCaRL VPT
Builds on: CLIP CoOp MaPLe AttriCLIP CLIP-Adapter PROOF
Closest papers by meaning
- Synthetic Data is an Elegant GIFT for Continual Vision-Language Models (CVPR 2025 · similarity 0.88)
- Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters (CVPR 2024 · similarity 0.88)
- Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning (AAAI 2024 · similarity 0.87)
- Convolutional Prompting meets Language Models for Continual Learning (CVPR 2024 · similarity 0.87)
- Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP (NeurIPS 2024 · similarity 0.87)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.