Research map / Continual learning / Continual fine-tuning of LLMs
Synthetic Data is an Elegant GIFT for Continual Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GIFT, a continual fine-tuning method for CLIP that uses Stable Diffusion-generated images (from ImageNet and downstream class names) with contrastive distillation, image-text alignment, and adaptive Fisher-weighted l2 consolidation to reduce forgetting of pre-training and downstream knowledge.
Problem
Continual fine-tuning of vision-language models like CLIP causes catastrophic forgetting of both downstream task knowledge and pre-training (zero-shot) knowledge, and original pre-training data is unavailable for replay.
Approach
Replace inaccessible replay data with diffusion-generated image-text pairs (prompted from ImageNet and past downstream class names), distill the CLIP similarity matrix contrastively from the previous model on these pairs, add an image-text alignment hard-target loss, and regularize with an adaptive Fisher-weighted l2 penalty whose Fisher information is recomputed during training from the distillat
Key results
- On MTIL Order I, GIFT reaches Transfer 69.3, Avg. 77.3, Last 86.0, outperforming ZSCL (68.1/75.4/83.6) and MoE-Adapter (68.9/76.7/85.0).
- On MTIL Order II, GIFT reaches Transfer 65.9, Avg. 75.7, Last 85.3, versus ZSCL at 64.2/74.5/83.4.
- GIFT with 1K synthetic images per task outperforms ZSCL which uses 100K ImageNet images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GIFT instead of the strongest baseline, MoE-Adapter | CLIP ViT-B/16 · MTIL Order I | Avg. | Helped |
| Use GIFT instead of the strongest baseline, MoE-Adapter | CLIP ViT-B/16 · MTIL Order II | Avg. | Helped |
| Add contrastive distillation (CD) to the l2 baseline (ablation) | CLIP ViT-B/16 · MTIL Order I | Avg.: +7.8 | Helped |
| Add image-text alignment (ITA) to the l2 baseline (ablation) | CLIP ViT-B/16 · MTIL Order I | Avg.: +13.6 | Helped |
| Add CD and ITA to the l2 baseline (ablation) | CLIP ViT-B/16 · MTIL Order I | Avg.: +13.9 | Helped |
| Add CD and AWC to the l2 baseline (ablation) | CLIP ViT-B/16 · MTIL Order I | Avg.: +14.3 | Helped |
| Add CD, ITA, and AWC to the l2 baseline (ablation) | CLIP ViT-B/16 · MTIL Order I | Avg.: +14.6 | Helped |
| Use contrastive distillation instead of feature-distance distillation (ablation) | CLIP ViT-B/16 · MTIL Order I | Avg.: 71.6 -> 76.6 | Helped |
| Use the last CLIP model rather than the initial CLIP as teacher (ablation) | CLIP ViT-B/16 · MTIL Order I | Last: 80.1 -> 85.0 | Helped |
| Set ITA scale to β = 0.25 rather than β = 0.0 (ablation) | CLIP ViT-B/16 · MTIL Order I | Avg.: 76.3 -> 76.6 | Helped |
| Generate 1K rather than 0 synthetic images per task (ablation) | GIFT · MTIL Order I | Avg.: 74.12 -> 76.13 | Helped |
| Use synthetic images rather than ImageNet images (1K samples) (ablation) | GIFT · MTIL Order I | Last: 84.0 -> 85.0 | Helped |
Surprising findings
- A plain l2 baseline without any distillation is strong on MTIL Order II, outperforming LwF, iCaRL, and WiSE-FT on Transfer, which the authors attribute to CLIP's broad, flat pre-training optimum.
- Synthetic images from downstream class names yield higher Avg. and Last than real ImageNet images, despite synthetic ImageNet-only data performing worse than real images.
Limitations
- Relies on quality and diversity of Stable Diffusion generations, which may contain noise compared with real images.
- Evaluated mainly on MTIL with CLIP ViT-B/16; generalization to other VLM architectures is not shown in the main text.
Experimental setup
Datasets: ImageNet Aircraft Caltech101 CIFAR100 DTD EuroSAT Flowers Food MNIST OxfordPet StanfordCars SUN397
Benchmarks: MTIL (Order I) MTIL (Order II)
Metrics: Transfer Avg. Last
Baselines: Continual Finetune l2 baseline LwF iCaRL LwF-VR WiSE-FT ZSCL MoE-Adapter
Builds on: ZSCL EWC CLIP WiSE-FT Stable Diffusion
Closest papers by meaning
- A Practitioner's Guide to Real-World Continual Multimodal Pretraining (NeurIPS 2024 · similarity 0.89)
- Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning (NeurIPS 2026 · similarity 0.89)
- Towards Self-Refinement of Vision-Language Models with Triangular Consistency (NeurIPS 2025 · similarity 0.89)
- Enhancing Visual Continual Learning with Language-Guided Supervision (CVPR 2024 · similarity 0.88)
- Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves (CVPR 2025 · similarity 0.88)
- Overcoming Generic Knowledge Loss with Selective Parameter Update (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.