Research map / Continual learning / Continual fine-tuning of LLMs
MagMax: Leveraging Model Merging for Seamless Continual Learning
Daniel Marczak, Bartlomiej Twardowski, Tomasz Trzcinski, Sebastian Cygert
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use sequential instead of independent fine-tuning before maximum magnitude merging (ablation) | MagMax · Class-incremental average (14 settings) | task-agnostic accuracy (%): 71.17 -> 74.50 | Improved |
| use sequential fine-tuning with TIES merging (ablation) | TIES · Class-incremental average (14 settings) | task-agnostic accuracy (%): 71.84 -> 74.37 | Improved |
| use sequential fine-tuning with Avg merging (ablation) | Avg · Class-incremental average (14 settings) | task-agnostic accuracy (%): 71.75 -> 73.05 | Improved |
| use sequential fine-tuning with Avg merging (ablation) | Avg · Eight-dataset benchmark | average accuracy (%): 67.54 -> 79.65 | Improved |
| use sequential fine-tuning with MagMax on dissimilar tasks (ablation) | MagMax · Eight-dataset benchmark | average accuracy (%): 74.14 -> 82.74 | Improved |
| add maximum magnitude merging to LwF | LwF · Class-incremental average (14 settings) | task-agnostic accuracy (%): 72.36 -> 73.85 | Improved |
| add maximum magnitude merging to EWC | EWC · Class-incremental average (14 settings) | task-agnostic accuracy (%): 70.79 -> 73.97 | Improved |
| start fine-tuning from first-task model theta_1 instead of pre-trained weights (ablation) | MagMax · Class-incremental average (14 settings) | task-agnostic accuracy (%): 71.17 -> 70.37 | Hurt |
| start from previous checkpoint theta_{t-1} instead of theta_1 (ablation) | MagMax · Class-incremental average (14 settings) | task-agnostic accuracy (%): 70.37 -> 74.50 | Improved |
| merge by maximum magnitude versus lower-magnitude selection for sequentially fine-tuned models | MagMax · CIFAR100 magnitude analysis | accuracy | Improved |
| use simple averaging or random mixing of independent fine-tunings instead of sophisticated merging | model merging baselines · Class-incremental average (14 settings) | task-agnostic accuracy (%): 71.75 / 71.78 vs TIES 71.84 | No clear effect |
| apply sequential fine-tuning to RandMix on fine-grained and Cars/20 settings (ablation) | RandMix · Cars /20 | task-agnostic accuracy (%): 64.95 -> 62.18 | Hurt |
Later papers that compared with this method
4 later accepted papers report MagMax as a baseline in their results tables; 3 of them report a better number on most of the results they share with it.
- Merge before Forget: A Single LoRA Continual Learning via Continual Merging (ICLR 2026 · better on 3 of 3 shared results)
- DSCA: Dynamic Subspace Concept Alignment for Lifelong VLM Editing (CVPR 2026 · better on 2 of 2 shared results)
- SCoT: Similarity-guided Conflict-aware Task Consolidation for Continual VQA (ECCV 2026 · better on 0 of 2 shared results)
- Mind the Gap: Preserving and Compensating for the Modality Gap in CLIP-Based Continual Learning (ICCV 2025 · better on 6 of 6 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning (NeurIPS 2025 · similarity 0.93)
- BECAME: Bayesian Continual Learning with Adaptive Model Merging (ICML 2025 · similarity 0.90)
- Model Sensitivity Aware Continual Learning (NeurIPS 2024 · similarity 0.90)
- Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.89)
- Learn more, but bother less: parameter efficient continual learning (NeurIPS 2024 · similarity 0.89)
- Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
