Research map / Continual learning / Continual fine-tuning of LLMs
LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LADA, a label-specific CLIP adapter that appends per-class memory units (k-means cluster centers, trained with distillation and GMM-based prototype augmentation) after the frozen CLIP image encoder, plus text-encoder fine-tuning via AdaptFormer, for cross-domain task-agnostic continual learning.
Problem
Continual learning with CLIP across multiple domains (X-TAIL) where task identity is unavailable at inference; existing prompt- and adapter-selection methods suffer misassignment errors and forward/backward forgetting.
Approach
Instead of partitioning parameters per task and selecting them at inference, append lightweight label-specific memory units (k-means centers per class, initialized from features and fine-tuned) after the frozen image encoder; old units are frozen, and old-class features are preserved via distilled GMM-augmented prototypes, so no gradients flow into CLIP's image encoder.
Key results
- 16-shot X-TAIL: LADA reaches 61.5 Transfer, 72.7 Average, 83.1 Last, versus Dual-RAIL at 71.3 Average and 82.3 Last.
- Full-shot X-TAIL: LADA reaches 61.9 Transfer, 75.2 Average, 86.9 Last, versus Primal-RAIL at 72.8 Average and 84.0 Last.
- Ablation: the adapter alone (BF + LADA) improves 16-shot Average from 70.9 to 72.3; DPT adds a further gain to 72.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LADA to BF (ablation) | CLIP ViT-B/16 · X-TAIL 16-shot | Average accuracy: 70.9 -> 72.3 | Helped |
| Add LADA to BF (ablation) | CLIP ViT-B/16 · X-TAIL 16-shot | Transfer accuracy: 59.4 -> 61.2 | Helped |
| Add LADA to BF (ablation) | CLIP ViT-B/16 · X-TAIL 16-shot | Last accuracy: 82.1 -> 83.0 | Helped |
| Add DPT to BF (ablation) | CLIP ViT-B/16 · X-TAIL 16-shot | Average accuracy: 70.9 -> 71.6 | Helped |
| Add DPT to BF (ablation) | CLIP ViT-B/16 · X-TAIL 16-shot | Transfer accuracy: 59.4 -> 59.9 | Helped |
| Add LADA to BF (ablation) | CLIP ViT-B/16 · X-TAIL full-shot | Average accuracy: 73.2 -> 74.9 | Helped |
| Add LADA to BF (ablation) | CLIP ViT-B/16 · X-TAIL full-shot | Transfer accuracy: 59.6 -> 61.1 | Helped |
| Add LADA to BF (ablation) | CLIP ViT-B/16 · X-TAIL full-shot | Last accuracy: 85.6 -> 86.8 | Helped |
| Add DPT to BF (ablation) | CLIP ViT-B/16 · X-TAIL full-shot | Average accuracy: 73.2 -> 74.4 | Helped |
| Increase prototype count from 1 to 16 at LADA dimension 16 (ablation) | LADA · X-TAIL full-shot | Transfer accuracy: 61.4 -> 62.3 | Helped |
| Increase prototype count from 1 to 16 at LADA dimension 16 (ablation) | LADA · X-TAIL full-shot | Average accuracy: 75.1 -> 74.8 | Hurt |
| Increase LADA dimension from 8 to 32 at 16 prototypes (ablation) | LADA · X-TAIL full-shot | Average accuracy: 75.1 -> 74.0 | Hurt |
Surprising findings
- On Flowers, Food and SUN397, LADA's Transfer accuracy exceeds that of vanilla zero-shot CLIP.
Limitations
- Dual-RAIL cannot be run in the full-shot setting due to storing all features and GPU memory limits.
- Relies on the X-TAIL benchmark with a fixed learning order for main results; the paper reports only two orders.
Experimental setup
Datasets: X-TAIL Aircraft Caltech101 DTD EuroSAT Flowers Food MNIST OxfordPet StanfordCars SUN397
Benchmarks: X-TAIL 16-shot X-TAIL full-shot
Metrics: Transfer Average Last
Baselines: LwF WiSE-FT ZSCL MoE-Adapters Primal-RAIL Dual-RAIL
Builds on: CLIP ZSCL MoE-Adapters AdaptFormer RAIL
Closest papers by meaning
- TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree (ICML 2025 · similarity 0.86)
- Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters (CVPR 2024 · similarity 0.85)
- Ranking-aware adapter for text-driven image ordering with CLIP (ICLR 2025 · similarity 0.85)
- C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models (NeurIPS 2026 · similarity 0.85)
- TiC-CLIP: Continual Training of CLIP Models (ICLR 2024 · similarity 0.85)
- Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.