Research map / Continual learning / Continual fine-tuning of LLMs
Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OPCM, a training-free sequential model merging method that projects each new task vector onto the subspace orthogonal to the current merged update via SVD and rescales with an adaptive factor, evaluated on CLIP-ViT image classification over 8/14/20 tasks.
Problem
Merging fine-tuned models when they arrive sequentially, where conventional merging needs all models simultaneously, memory grows linearly with the number of models, and merge order affects results.
Approach
Project each incoming task vector onto the orthogonal complement of the current merged update's top singular subspace (SVD-based, threshold alpha), then rescale the merged model by an adaptive factor (approximately sqrt(t)) so its distance from the pre-trained model stays stable, keeping only the merged model and the base model in memory.
Key results
- On ViT-B/32 with 20 tasks, OPCM reaches 65.7% average accuracy vs 60.6% for continual Task Arithmetic and 59.9% for continual Ties-Merging (10 shuffled orders).
- On ViT-L/14 with 8 tasks, OPCM achieves 87.0% average accuracy vs 82.1% for continual Task Arithmetic.
- OPCM std across task orders is small (e.g., ±0.2 to ±0.5 for accuracy), whereas continual Ties-Merging varies up to ±10.2 on 8 tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OPCM instead of continual task arithmetic | CLIP-ViT-B/32 merging · 8 tasks | ACC: 67.5±0.0 -> 75.5±0.5 | Helped |
| Use OPCM instead of continual task arithmetic | CLIP-ViT-B/32 merging · 14 tasks | ACC: 66.5±0.0 -> 71.9±0.3 | Helped |
| Use OPCM instead of Average (SWA) | CLIP-ViT-B/32 merging · 20 tasks | ACC: 61.1±0.0 -> 65.7±0.2 | Helped |
| Use OPCM instead of continual task arithmetic | CLIP-ViT-B/16 merging · 8 tasks | ACC: 77.1±0.0 -> 81.8±0.3 | Helped |
| Use OPCM instead of continual task arithmetic | CLIP-ViT-B/16 merging · 14 tasks | ACC: 70.9±0.0 -> 77.1±0.5 | Helped |
| Use OPCM instead of Average (SWA) | CLIP-ViT-B/16 merging · 20 tasks | ACC: 64.8±0.0 -> 70.3±0.2 | Helped |
| Use OPCM instead of continual task arithmetic | CLIP-ViT-L/14 merging · 8 tasks | ACC: 82.1±0.0 -> 87.0±0.4 | Helped |
| Use OPCM instead of continual Ties-Merging | CLIP-ViT-L/14 merging · 14 tasks | ACC: 78.0±0.6 -> 83.5±0.2 | Helped |
| Use OPCM instead of Average (SWA) | CLIP-ViT-L/14 merging · 20 tasks | ACC: 71.1±0.0 -> 76.0±0.2 | Helped |
| Vary the projection threshold α (ablation) | OPCM on ViT-B/32 · 8, 14, and 20 task sets | ACC | Mixed |
| Increase model size from ViT-B/32 to ViT-L/14 | OPCM · 8-task set | BWT: -6.3% -> -2.6% | Helped |
Surprising findings
- Continual Ties-Merging has higher (less negative) BWT than OPCM on 14 and 20 tasks for ViT-B/32 and ViT-L/14 in several settings, so OPCM does not dominate on backward transfer across all configurations.
- Continual fine-tuning on ViT-B/32 with 20 tasks reaches only 62.6% average accuracy, below the simple weight averaging baseline in some settings.
Limitations
- Evaluated only on vision classification with CLIP-ViT models; extensions to language and multimodal models are untested.
- Relies on approximate orthogonality of task vectors to justify the sqrt(t) scaling; the adaptive variant is an empirical choice.
- Hyperparameter alpha (projection threshold) and the baseline merging coefficients were tuned on the evaluation setting; BWT is not uniformly best.
Experimental setup
Datasets: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD Flowers102 PCAM FER2013 OxfordIIITPet
Benchmarks: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD Flowers102 PCAM FER2013 OxfordIIITPet
Metrics: average accuracy backward transfer
Baselines: Continual Task Arithmetic Continual Ties-Merging Simple weight averaging (SWA) Continual fine-tuning
Builds on: task arithmetic TIES-Merging model soups FusionBench
Closest papers by meaning
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.91)
- No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces (ICML 2025 · similarity 0.90)
- Dynamic Model Merging Made Slim (NeurIPS 2026 · similarity 0.90)
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.90)
- Rethinking Layer-wise Model Merging through Chain of Merges (NeurIPS 2026 · similarity 0.90)
- EMR-Merging: Tuning-Free High-Performance Model Merging (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.