Research map / Continual learning / Continual fine-tuning of LLMs
Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models
Yu-Chu Yu, Chi-Pin Huang, Jr-Jen Chen, Kai-Po Chang, Yung-Hsuan Lai, Fu-En Yang, Yu-Chiang Frank Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill from both the pre-trained and most recent fine-tuned model, selected by dual-teacher discrepancy (ablation) | Select and Distill · MTIL sequence S1 | average accuracy: 81.35 -> 84.48 (vs g0 only), 83.61 -> 84.48 (vs g_{k-1} only) | Improved |
| add distillation from the most recent fine-tuned teacher to pre-trained-teacher distillation (ablation) | Select and Distill · MTIL sequence S1 | catastrophic forgetting: 5.26 -> 1.70 | Improved |
| add distillation from the pre-trained teacher to recent-teacher distillation (ablation) | Select and Distill · MTIL sequence S1 | zero-shot degradation: 3.36 -> 1.55 | Improved |
| Select and Distill instead of ZSCL | CLIP continual learning · MTIL mean | catastrophic forgetting: 3.64 -> 1.20 | Improved |
| Select and Distill instead of MoE-Adapters on zero-shot degradation | CLIP continual learning · MTIL mean | zero-shot degradation: 2.17 -> 1.96 | Improved |
| use class-incremental (open-set) evaluation instead of task-incremental | Select and Distill · MCIL vs MTIL mean | average accuracy: 84.92 -> 83.76 | Hurt |
| increase selection threshold delta from 0.1/0.2 to 0.3 (ablation) | eta selection function · MTIL sequence S1 | average accuracy: 84.48 -> 84.03 (delta 0.3, gamma 1/6) | Hurt |
| increase reference set size from 10k to 200k ImageNet images (ablation) | Select and Distill · MTIL sequence S1 | zero-shot degradation: 2.12 -> 1.11 | Improved |
| use Conceptual Captions 12M as the reference dataset instead of ImageNet (100k) (ablation) | Select and Distill · MTIL sequence S1 | average accuracy: 84.48 -> 84.48 | No clear effect |
Closest papers by meaning
- Synthetic Data is an Elegant GIFT for Continual Vision-Language Models (CVPR 2025 · similarity 0.89)
- Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners (CVPR 2024 · similarity 0.89)
- Masking Teacher and Reinforcing Student for Distilling Vision-Language Models (CVPR 2026 · similarity 0.89)
- Stabilizing Zero-Shot Prediction: A Novel Antidote to Forgetting in Continual Vision-Language Tasks (NeurIPS 2024 · similarity 0.89)
- Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
