AI Paper Atlas

Research map / Continual learning / Continual fine-tuning of LLMs

Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models

Yu-Chu Yu, Chi-Pin Huang, Jr-Jen Chen, Kai-Po Chang, Yung-Hsuan Lai, Fu-En Yang, Yu-Chiang Frank Wang

ECCV 2024 · Continual learning › Continual fine-tuning of LLMs · approach: Synthetic data and merging for CLIP · 11 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
distill from both the pre-trained and most recent fine-tuned model, selected by dual-teacher discrepancy (ablation)Select and Distill · MTIL sequence S1average accuracy: 81.35 -> 84.48 (vs g0 only), 83.61 -> 84.48 (vs g_{k-1} only)Improved
add distillation from the most recent fine-tuned teacher to pre-trained-teacher distillation (ablation)Select and Distill · MTIL sequence S1catastrophic forgetting: 5.26 -> 1.70Improved
add distillation from the pre-trained teacher to recent-teacher distillation (ablation)Select and Distill · MTIL sequence S1zero-shot degradation: 3.36 -> 1.55Improved
Select and Distill instead of ZSCLCLIP continual learning · MTIL meancatastrophic forgetting: 3.64 -> 1.20Improved
Select and Distill instead of MoE-Adapters on zero-shot degradationCLIP continual learning · MTIL meanzero-shot degradation: 2.17 -> 1.96Improved
use class-incremental (open-set) evaluation instead of task-incrementalSelect and Distill · MCIL vs MTIL meanaverage accuracy: 84.92 -> 83.76Hurt
increase selection threshold delta from 0.1/0.2 to 0.3 (ablation)eta selection function · MTIL sequence S1average accuracy: 84.48 -> 84.03 (delta 0.3, gamma 1/6)Hurt
increase reference set size from 10k to 200k ImageNet images (ablation)Select and Distill · MTIL sequence S1zero-shot degradation: 2.12 -> 1.11Improved
use Conceptual Captions 12M as the reference dataset instead of ImageNet (100k) (ablation)Select and Distill · MTIL sequence S1average accuracy: 84.48 -> 84.48No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy