Research map / Continual learning / Continual fine-tuning of LLMs
Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PriViLege, which adapts a pre-trained ViT-B/16 for few-shot class incremental learning using pre-trained knowledge tuning, an entropy-based divergence loss, and semantic knowledge distillation from BERT class-name embeddings.
Problem
Few-shot class incremental learning (FSCIL) suffers from catastrophic forgetting and overfitting, and prior work relies on shallow ResNet-18 models that limit knowledge transfer; large pre-trained ViTs are hard to use because fine-tuning forgets and freezing blocks domain learning.
Approach
Selectively tune the first two ViT layers plus a base prompt (prefix-tuned and scaled by lightweight modulation prompts from MSA/MLP features) and a vision-language prompt, while freezing the model after the base session; add an entropy-based divergence loss to sharpen the vision token and distill BERT class-name embeddings into the language token.
Key results
- On CUB200, PriViLege reaches 77.50% average accuracy and 75.08% last-session accuracy, about +9.38% last-session gain over CEC.
- On CIFAR-100, PriViLege reports about +20.58% last-session accuracy over WaRP (88.08% average).
- Ablation on CUB200: PKT alone raises last-session accuracy from 3.79% (fine-tuning baseline) to 70.81%; adding LSKD and LED further improves it to 75.08%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PKT instead of fine-tuning the pre-trained ViT with prototype classifier ψ (ablation) | ViT-B/16 FSCIL · CUB200 | ALast accuracy: 3.79±1.47 -> 70.81±0.76 | Helped |
| add LED to PKT (ablation) | PriViLege · CUB200 | ALast accuracy: 70.81±0.76 -> 72.70±0.45 | Helped |
| add LSKD to PKT (ablation) | PriViLege · CUB200 | ALast accuracy: 70.81±0.76 -> 73.44±0.40 | Helped |
| add LSKD to PKT + LED (ablation) | PriViLege · CUB200 | ALast accuracy: 72.70±0.45 -> 75.08±0.52 | Helped |
| tune 2 layers rather than 0 layers (ablation) | PKT · CUB200 | ALast accuracy: 60.19±1.11 -> 70.81±0.76 | Helped |
| tune 5 rather than 2 layers (ablation) | PKT · CUB200 | ALast accuracy: 70.81±0.76 -> 68.52±0.84 | Hurt |
| tune 7 rather than 2 layers (ablation) | PKT · CUB200 | ALast accuracy: 70.81±0.76 -> 63.15±2.73 | Hurt |
| tune 10 rather than 2 layers (ablation) | PKT · CUB200 | ALast accuracy: 70.81±0.76 -> 57.78±2.30 | Hurt |
| tune 12 rather than 2 layers (ablation) | PKT · CUB200 | ALast accuracy: 70.81±0.76 -> 56.02±1.47 | Hurt |
| use PriViLege instead of CEC | ViT-B/16 FSCIL · CUB200 | ALast accuracy: 65.70±8.03 -> 75.08±0.52 | Helped |
| use PriViLege instead of WaRP | ViT-B/16 FSCIL · CIFAR-100 | ALast accuracy: 65.48±1.87 -> 86.06±0.32 | Helped |
| use PriViLege instead of CEC | ViT-B/16 FSCIL · miniImageNet | ALast accuracy: 80.74±7.51 -> 94.10±0.13 | Helped |
Surprising findings
- Directly applying prompt-based CIL methods (L2P, DualPrompt) to FSCIL underperforms existing shallow-network FSCIL methods.
- Tuning more than two ViT layers consistently degrades base and last-session accuracy.
Limitations
- Relies on a large base session with many classes; the authors note future work is needed for FSCIL without a base session.
- Evaluation limited to three image classification benchmarks with a ViT-B/16 backbone and CLIP as a secondary check.
Experimental setup
Datasets: CIFAR-100 miniImageNet CUB200
Benchmarks: CUB200 CIFAR-100 miniImageNet
Metrics: base session accuracy last session accuracy average accuracy
Baselines: CEC WaRP NC-FSCIL L2P DualPrompt Fine-Tuning + Proto
Builds on: L2P DualPrompt CEC prefix tuning prompt tuning
Closest papers by meaning
- SEC-Prompt:SEmantic Complementary Prompting for Few-Shot Class-Incremental Learning (CVPR 2025 · similarity 0.88)
- Tripartite Weight-Space Ensemble for Few-Shot Class-Incremental Learning (CVPR 2025 · similarity 0.88)
- Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- Masking Teacher and Reinforcing Student for Distilling Vision-Language Models (CVPR 2026 · similarity 0.87)
- Compositional Few-Shot Class-Incremental Learning (ICML 2024 · similarity 0.87)
- Making Large Vision Language Models to Be Good Few-Shot Learners (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.