Research map / Continual learning / Domain and prompt-based
Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RAIL, a recursive ridge-regression adapter on frozen CLIP with a training-free zero-shot fusion module, plus the X-TAIL task-agnostic cross-domain continual learning setting.
Problem
Continual learning with vision-language models must avoid forgetting incremental domains and preserve CLIP's zero-shot ability on unseen domains, without reference datasets or domain-identity hints at test time.
Approach
Replace gradient-based adapter training with recursive ridge regression (primal with random hidden-layer projection, dual with RBF kernel) that provably equals joint training, and fuse with frozen CLIP zero-shot logits via an ID/OOD gate so unseen-domain predictions are never degraded.
Key results
- Dual-RAIL reaches 71.9% Average and 82.4% Last on X-TAIL (alphabetical order), versus 63.0% Average for MoE-Adapter, and 62.4% Transfer equal to zero-shot-preserving values.
- Primal-RAIL achieves 69.4% Average and 77.2% Last on 5-shot MTIL, outperforming MoE (68.9 Average-style transfer and 76.1 Last).
- Fusion strategy with beta=0.8 improves Last accuracy by 1.9% (primal) and 1.3% (dual) over pure RAIL-Adapter.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Dual-RAIL instead of the strongest baseline, MoE-Adapter | CLIP ViT-B/16 · X-TAIL (10 domains) | Average accuracy: 63.0 -> 71.9 | Helped |
| Use Dual-RAIL instead of the strongest baseline, MoE-Adapter | CLIP ViT-B/16 · X-TAIL (10 domains) | Last accuracy: 70.5 -> 82.4 | Helped |
| Use Primal-RAIL instead of the strongest baseline, iCaRL | CLIP ViT-B/16 · X-TAIL (10 domains) | Transfer accuracy: 61.7 -> 62.4 | Helped |
| Use Dual-RAIL instead of the strongest baseline, MoE | CLIP ViT-B/16 · 5-shot MTIL | Average accuracy: 71.4 -> 72.3 | Helped |
| Increase the Primal-RAIL RHL dimension from 1k to 10k (ablation) | Primal-RAIL · X-TAIL (10 domains) | Last accuracy: 73.6% to 79.0% | Helped |
| Increase the Primal-RAIL RHL dimension from 10k to 15k (ablation) | Primal-RAIL · X-TAIL (10 domains) | Last accuracy: 79.0% to 79.1% | Helped |
| Increase the Primal-RAIL RHL dimension from 15k to 20k (ablation) | Primal-RAIL · X-TAIL (10 domains) | Last accuracy: 79.1% -> 79.1% | No clear effect |
| Use one-hot labels rather than text embeddings as regression targets (ablation) | Dual RAIL-Adapter · X-TAIL (10 domains) | Last accuracy: +3.8% | Helped |
| Use one-hot labels rather than text embeddings as regression targets (ablation) | Dual RAIL-Adapter on Aircraft · X-TAIL | Last accuracy: +7.5% | Helped |
| Use RAIL-Fusion instead of a zero-shot domain indicator with multiple domain-specific classifiers (ablation) | Dual RAIL-Adapter · X-TAIL (10 domains) | Last accuracy: +7.5% | Helped |
| Set fusion ratio β to 0.8 instead of β=1 (ablation) | Primal-RAIL · X-TAIL (10 domains) | Last accuracy: +1.9% | Helped |
| Set fusion ratio β to 0.8 instead of β=1 (ablation) | Dual-RAIL · X-TAIL (10 domains) | Last accuracy: +1.3% | Helped |
Surprising findings
- Using one-hot regression targets outperforms text-embedding targets by an average of 3.8% in Last accuracy.
- Text-embedding regression targets are suboptimal, with Aircraft Last accuracy 7.5% lower, attributed to unhelpful class names.
Limitations
- The pre-trained VLM encoder stays frozen, so feature extraction cannot improve during incremental learning.
- Evaluated only on image classification; extension to other downstream tasks such as segmentation is untested.
Experimental setup
Datasets: Aircraft Caltech101 DTD EuroSAT Flowers Food101 MNIST OxfordPet StanfordCars SUN397 CIFAR-100
Benchmarks: X-TAIL (10 domains) MTIL (5-shot, 11 datasets)
Metrics: Transfer Average Last accuracy
Baselines: LwF WiSE-FT iCaRL ZSCL MoE-Adapter LwF-VR Fine-tune Zero-shot
Builds on: CLIP ZSCL MoE-Adapters Woodbury matrix identity Multi-domain Task-Incremental Learning
Closest papers by meaning
- Revisiting Continuity of Image Tokens for Cross-domain Few-shot Learning (ICML 2025 · similarity 0.89)
- Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters (CVPR 2024 · similarity 0.89)
- Approximate Domain Unlearning for Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- Synthetic Data is an Elegant GIFT for Continual Vision-Language Models (CVPR 2025 · similarity 0.88)
- Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners (CVPR 2024 · similarity 0.88)
- Stabilizing Zero-Shot Prediction: A Novel Antidote to Forgetting in Continual Vision-Language Tasks (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.