Research map / Architectures and efficiency / Mixture of experts
How to Merge Your Multimodal Models Over Time?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TIME, a framework for temporal model merging of CLIP experts over sequential tasks, finding that EMA-style initialization and deployment (Best-in-TIME) matter far more than the choice of merging technique.
Problem
Existing model merging assumes all expert models are available simultaneously, but in practice new tasks and domains arrive over time, so experts must be integrated sequentially without a fixed set.
Approach
Decomposes temporal merging into three axes (initialization, deployment, merging technique) and shows that EMA-style initialization and deployment over sequentially trained experts (Best-in-TIME) outperforms offline merging and sequential fine-tuning, with merge technique mattering little.
Key results
- Offline merging techniques (any of them) fail to add new knowledge over time and underperform a simple replay baseline (replay reaches 59.1% vs best-offline 54.6%).
- Best-in-TIME (EMA initialization and EMA deployment) gives the best knowledge accumulation and retention balance; all seven merge techniques perform similarly within it.
- Best-in-TIME scales with model size from ViT-S/16 to ViT-g/14 and approaches the multitask upper bound as compute grows.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use temporal replay rather than offline merging | Model merging on ViT-B/16 CLIP · FoMo-in-Flux | geometric mean of AKA and AZS: 54.6% -> 59.1% | Helped |
| Add replay of previous-task data | Offline model merging · FoMo-in-Flux | geometric mean of AKA and AZS: 54.6% -> 58.2% | Helped |
| Add recency-biased weighting | Offline merging with replay · FoMo-in-Flux | geometric mean of AKA and AZS: 58.2% -> 58.9% | Helped |
| Use EMA for both initialization and deployment | TIME · FoMo-in-Flux | knowledge accumulation and zero-shot retention | Helped |
| Replace weight averaging with other tested merging techniques while retaining Best-in-TIME initialization and deployment (ablation) | Best-in-TIME · FoMo-in-Flux | knowledge accumulation | No clear effect |
| Use offline merging without temporal adaptation | Temporal model merging · FoMo-in-Flux | knowledge accumulation | Hurt |
| Scale model size | Best-in-TIME · FoMo-in-Flux | knowledge accumulation and retention | Helped |
| Increase the number of tasks to 50 or 100 | Best-in-TIME · FoMo-in-Flux | knowledge accumulation | Helped |
| Increase compute budget per task | Best-in-TIME · FoMo-in-Flux | geometric mean of AKA and AZS | Helped |
Surprising findings
- Standard offline merging, regardless of merge technique, underperforms even a simple sequential replay baseline on the temporal setting.
- The choice of sophisticated merging technique (TIES, DARE, LiNeS, etc.) provides at most marginal benefit over simple weight averaging for temporal merging.
Limitations
- Evaluation restricted to CLIP ViT models on the FoMo-in-Flux continual pretraining benchmark.
- Training strategy kept fixed; only merging, initialization and deployment were varied.
Experimental setup
Datasets: FoMo-in-Flux LAION-2B LAION-400M
Benchmarks: FoMo-in-Flux
Metrics: Knowledge Accumulation (AKA) Zero-Shot Retention (AZS) geometric mean of AKA and AZS
Baselines: sequential fine-tuning (replay) offline model merging (averaging, task arithmetic, MagMax, TIES, DARE-TIES, Brea zero-shot base model multitask training
Builds on: FoMo-in-Flux Task Arithmetic TIES DARE Model Stock
Closest papers by meaning
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.88)
- OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.88)
- Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.87)
- A Practitioner's Guide to Real-World Continual Multimodal Pretraining (NeurIPS 2024 · similarity 0.87)
- Dynamic Model Merging Made Slim (NeurIPS 2026 · similarity 0.87)
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.