Research map / Continual learning / Continual fine-tuning of LLMs
TiC-CLIP: Continual Training of CLIP Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TiC-CLIP, time-stamped web-scale continual training benchmarks (TiC-DataComp, TiC-YFCC, TiC-RedCaps) and shows that warm-starting CLIP from the last checkpoint while replaying all past data matches retraining from scratch at about 2.5-4x less compute.
Problem
Keeping CLIP-style vision-language foundation models up to date with time-evolving web data is costly, and there were no large-scale time-continual benchmarks or baselines for continually training CLIP.
Approach
Attach crawl/upload timestamps to existing web image-text datasets to create time-continual splits, build dynamic retrieval and classification evaluations across years, and benchmark simple replay-with-warm-start continual training against retraining from scratch under a fixed per-step compute budget.
Key results
- Cumulative-All reaches within 1% of Oracle on dynamic retrieval tasks on TiC-DataComp while using about 4x less compute (and 2.5x less on TiC-YFCC and TiC-RedCaps).
- On TiC-DataComp (L), Cumulative-All gets 48.0 average static accuracy vs 50.4 for Oracle, with compute 2.7e19 vs 1.1e20 MACs.
- OpenAI CLIP loses about 8% zero-shot retrieval accuracy on 2021-2022 data compared with recent OpenCLIP models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replay all previous data and warm-start from the last checkpoint instead of training from scratch | Cumulative-All vs Oracle · TiC-YFCC | Average over 28 datasets: 29.0 -> 29.6 | No clear effect |
| Replay all previous data and warm-start from the last checkpoint instead of training from scratch | Cumulative-All vs Oracle · TiC-RedCaps | Average over 28 datasets: 32.3 -> 31.7 | No clear effect |
| Replay all previous data and warm-start from the last checkpoint instead of training from scratch | Cumulative-All vs Oracle · TiC-DataComp (M) | Average over 28 datasets: 30.8 -> 30.0 | No clear effect |
| Replay all previous data rather than train only on new data | TiC-DataComp (L) · TiC-DataComp (L) | Backward transfer: 52.6 -> 62.1 | Helped |
| Add replay of a buffer split equally among previous years instead of replaying all past data (ablation) | Cumulative-All · TiC-DataComp (M) | Average over 28 datasets: 30.0 -> 28.0 | Hurt |
| Add replay of a buffer halved at each step instead of replaying all past data (ablation) | Cumulative-All · TiC-DataComp (M) | Average over 28 datasets: 30.0 -> 28.8 | Hurt |
| Do not use warmup on subsequent training runs (ablation) | Cumulative-All · TiC-DataComp (L) | ImageNet accuracy: 47.6 -> 48.9 | Helped |
| Do not use warmup on subsequent training runs (ablation) | Cumulative-All · TiC-DataComp (L) | ID retrieval performance: 53.0 -> 57.3 | Helped |
| Add warmup to the first training run (ablation) | Cumulative-All · TiC-DataComp (L) | ImageNet accuracy at first time step: 24.1 -> 29.9 | Helped |
| Keep the same maximum learning rate across runs rather than reduce it to 0.00005 or 0.0001 (ablation) | Cumulative-All · TiC-DataComp (M) | ImageNet accuracy: 16.3 -> 24.0; 19.0 -> 24.0 | Helped |
| Use maximum learning rate 0.0005 instead of 0.00025 (ablation) | Cumulative-All · TiC-DataComp (M) | ImageNet accuracy: 24.0 -> 10.1 | Hurt |
| Use Const-Cosine instead of cyclic cosine learning-rate schedule (ablation) | Cumulative-All · TiC-YFCC | ImageNet accuracy: 29.3 -> 32.8 | Helped |
Surprising findings
- Not using learning-rate warmup on subsequent continual training runs is strictly better than using warmup, while warmup is needed for the first step.
- Pure sequential training without replay strongly forgets on static and backward-transfer tasks at large scale despite strong forward transfer.
Limitations
- Timestamps are at the granularity of months with noisy estimates and experiments use yearly granularity.
- Replay of all past data requires storing the full data history, which may be restricted by data expiration policies.
- Continual methods compared are limited; EWC, GEM, and IMM are excluded or only partly covered due to extra compute.
Experimental setup
Datasets: TiC-DataComp TiC-YFCC TiC-RedCaps DataComp CommonPool YFCC15M RedCaps ImageNet
Benchmarks: ImageNet ImageNet distribution shifts Flickr30k 28 zero-shot classification and retrieval tasks TiC-DataComp-Retrieval TiC-DataComp-Net
Metrics: zero-shot accuracy Recall@1 backward transfer forward transfer in-domain performance
Baselines: Oracle (retrain from scratch) Restart Sequential Patching LwF Cumulative-Exp Cumulative-Equal
Builds on: CLIP DataComp Learning without Forgetting Patching CLOC
Closest papers by meaning
- Demystifying CLIP Data (ICLR 2024 · similarity 0.88)
- What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspective (NeurIPS 2026 · similarity 0.87)
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.87)
- A Practitioner's Guide to Real-World Continual Multimodal Pretraining (NeurIPS 2024 · similarity 0.86)
- Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning (AAAI 2024 · similarity 0.86)
- CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.