Research map / Continual learning / Continual fine-tuning of LLMs
Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Revisits parameter-efficient tuning for class-incremental learning on a pre-trained ViT, finds unconstrained incremental adapter tuning works best, and retrains a unified classifier using semantically shifted prototypes (SSIAT).
Problem
Class-incremental learning with pre-trained vision transformers suffers from catastrophic forgetting; prior work relies on prompt pools, full fine-tuning, or first-session adaptation, and classifiers trained per session are misaligned.
Approach
Incrementally tune a single shared adapter (no parameter-update constraints, no adapter pool) with a local cosine classifier, then estimate semantic shift of stored prototypes without past samples (SDC-style) and retrain a unified classifier by sampling features from Gaussians built on updated prototypes.
Key results
- On ImageNet-A (10 sessions), SSIAT reaches 70.83% average accuracy with 1.19M tunable parameters, versus 68.88% for SLCA with 86M parameters.
- On CUB200, SSIAT achieves 93.00% average accuracy, about 2.3 points above SLCA (90.77%).
- On ImageNet-R, SSIAT reaches 79.38% last-session accuracy, above SLCA's 79.35% with comparable average (83.63% vs 83.29%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SSIAT instead of SLCA | ViT-B/16 continual learning · Split-ImageNetR | A_Avg: 83.29±0.46 -> 83.63±0.43 | Helped |
| Use SSIAT instead of SLCA | ViT-B/16 continual learning · Split-ImageNetA | A_Avg: 68.88±2.31 -> 70.83±1.63 | Helped |
| Use SSIAT instead of SLCA | ViT-B/16 continual learning · CUB200 | A_Avg: 90.77±0.79 -> 93.00±0.90 | Helped |
| Use SSIAT instead of SLCA | ViT-B/16 continual learning · CIFAR100 | A_Avg: 94.29±0.92 -> 94.35±0.60 | No clear effect |
| Use SSIAT instead of SLCA | ViT-B/16 continual learning · ImageNetR (20 sessions) | A_Avg: 79.92±1.29 -> 82.30±0.36 | Helped |
| Use SSIAT instead of SLCA | ViT-B/16 continual learning · ImageNetA (20 sessions) | A_Avg: 56.35±7.09 -> 68.45±1.92 | Helped |
| Replace adapter tuning with SSF (ablation) | CIFAR100 incremental adapter baseline · CIFAR100 | average session accuracy: 91.52 -> 83.03 | Hurt |
| Replace adapter tuning with VPT-deep (ablation) | CIFAR100 incremental adapter baseline · CIFAR100 | average session accuracy: 91.52 -> 61.72 | Hurt |
| Replace adapter tuning with VPT-shallow (ablation) | CIFAR100 incremental adapter baseline · CIFAR100 | average session accuracy: 91.52 -> 88.72 | Hurt |
| Add classifier alignment to the linear classifier (ablation) | SSIAT · ImageNetA | A_Avg: 58.99 -> 63.10 | Helped |
| Add semantic shift estimation to classifier alignment (ablation) | SSIAT with linear classifier · ImageNetA | A_Avg: 63.10 -> 63.55 | Helped |
| Add classifier alignment to the cosine classifier (ablation) | SSIAT · ImageNetA | A_Avg: 66.56 -> 68.68 | Helped |
Surprising findings
- Unconstrained incremental adapter tuning without any adapter expansion outperforms prompt-based and SSF tuning in class-incremental settings.
- Traditional expansion-based methods such as Foster and Der are competitive on ImageNet-R but perform poorly on ImageNet-A.
Limitations
- Relies on prototype statistics and Gaussian sampling, so it depends on Gaussian assumptions about class feature distributions.
- Semantic shift is estimated from current-task samples only, which is an approximation of old-class drift.
Experimental setup
Datasets: CIFAR100 CUB200 ImageNet-R ImageNet-A VTAB
Benchmarks: CIFAR100 CUB200 ImageNet-R ImageNet-A VTAB
Metrics: last-session accuracy average incremental accuracy
Baselines: SLCA Adam-adapter Adam-ssf Adam-prompt LAE L2P ADA DualPrompt CODAPrompt iCaRL Foster Der
Builds on: AdaptFormer SDC (semantic drift compensation) SLCA ALICE LAE
Closest papers by meaning
- Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation (NeurIPS 2024 · similarity 0.82)
- One Size Fits All for Semantic Shifts: Adaptive Prompt Tuning for Continual Learning (ICML 2024 · similarity 0.80)
- Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves (CVPR 2025 · similarity 0.80)
- Minimal Interaction Seperated Tuning: A New Paradigm for Visual Adaptation (CVPR 2025 · similarity 0.80)
- Synthetic Data is an Elegant GIFT for Continual Vision-Language Models (CVPR 2025 · similarity 0.80)
- Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.