Research map / Continual learning / Continual fine-tuning of LLMs
A Practitioner's Guide to Real-World Continual Multimodal Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FoMo-in-Flux, a compute-constrained continual multimodal pretraining benchmark of 63 datasets, and uses it to study methods, learning-rate schedules, model/compute scaling, and data stream orderings and mixtures for minor CLIP updates.
Problem
Foundation vision-language models become outdated, and existing continual pretraining research covers either infrequent large-scale updates or frequent sample-level edits, leaving the practical 'minor update' regime under-studied.
Approach
A controlled, concept-level data stream benchmark (FoMo-in-Flux) with compute budgets measured by Memory-Adjusted FLOPs (MAFs), enabling fair comparison of continual finetuning, PEFT, continual learning regularizers, and model merging, plus meta learning-rate schedules across long update cycles.
Key results
- Model merging (EMA-merge, w=0.9) achieves strong knowledge accumulation with maximal zero-shot retention, and is the only method that scales log-linearly with compute budget.
- Autoregressive rsqrt meta learning-rate schedule achieves strong AKA gains while greatly increasing retained zero-shot knowledge, beyond the linear trade-off line.
- Scaling ViT from S/16 to g/14 improves zero-shot retention while knowledge accumulation gain stays roughly constant (~8-10%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use naive contrastive fine-tuning instead of retention-focused parameter-selective tuning | CLIP continual pretraining · FoMo-in-Flux | AKA | Helped |
| Use naive contrastive fine-tuning instead of retention-focused parameter-selective tuning | CLIP continual pretraining · FoMo-in-Flux | AZS | Hurt |
| Use EWC regularization instead of standard fine-tuning | CLIP continual pretraining · FoMo-in-Flux | AZS | Helped |
| Use EWC regularization instead of standard fine-tuning | CLIP continual pretraining · FoMo-in-Flux | AKA | Hurt |
| Use SI regularization instead of standard fine-tuning | CLIP continual pretraining · FoMo-in-Flux | AKA | No clear effect |
| Merge updated weights with prior weights using EMA-merge or zero-shot merge | CLIP continual pretraining · FoMo-in-Flux | AZS | Helped |
| Increase the old-new weight mixing coefficient (ablation) | Model merging · FoMo-in-Flux | AZS | Helped |
| Increase the old-new weight mixing coefficient (ablation) | Model merging · FoMo-in-Flux | AKA | Hurt |
| Switch from cosine to reciprocal-square-root learning-rate scheduling (ablation) | CLIP continual fine-tuning · FoMo-in-Flux | AKA | No clear effect |
| Use a meta learning-rate schedule across continual update tasks | CLIP continual fine-tuning · FoMo-in-Flux | AZS | Helped |
| Use a cosine-derived meta schedule instead of task-independent cosine scheduling (ablation) | CLIP continual fine-tuning · FoMo-in-Flux | AKA | Hurt |
| Scale the vision transformer from ViT-B/16 to ViT-L/14 (ablation) | CLIP continual pretraining · FoMo-in-Flux | AZS: ≈3.2% -> ≈0.8% | Helped |
Surprising findings
- Traditional regularization methods EWC and SI give little benefit over naive finetuning at this scale, with EWC offering very limited knowledge accumulation.
- Final performance endpoints are similar across different concept-level stream orderings, despite different intermediate trajectories.
Limitations
- Findings are tied to the chosen adaptation and evaluation datasets, which may not cover all subdomains.
- Method hyperparameters were taken from default ranges of original publications, so conclusions depend on those being near-optimal.
- Scaling studies limited to ViT sizes up to g/14 and a single compute reference point.
Experimental setup
Datasets: FoMo-in-Flux LAION-400M CC-12M CC-3M DataComp-Small ImageNet MSCOCO Flickr30k Birdsnap CUB200-2011 Food101 Places365
Benchmarks: FoMo-in-Flux
Metrics: Knowledge Accumulation (AKA) Zero-Shot Retention (AZS) Memory-Adjusted FLOPs (MAFs) recall@5 for retrieval
Baselines: full fine-tuning LNFit BitFit LoRA VeRA DoRA GaLore EWC SI
Builds on: CLIP LoRA EWC Synaptic Intelligence DataComp
Closest papers by meaning
- Synthetic Data is an Elegant GIFT for Continual Vision-Language Models (CVPR 2025 · similarity 0.89)
- Context-Aware Multimodal Pretraining (CVPR 2025 · similarity 0.89)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.88)
- MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models (ICML 2026 · similarity 0.88)
- Continual Multimodal Contrastive Learning (NeurIPS 2025 · similarity 0.88)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.