Research map / Architectures and efficiency / Pruning and efficient tuning
Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks
MohammadReza Davari, Eugene Belilovsky
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Breadcrumbs sparse masking (both tails) to task vectors instead of plain Task Arithmetic | multi-task merging of CLIP · 8 vision tasks (CLIP) | avg normalized accuracy: 77.66 -> 83.35 | Improved |
| use magnitude-based masking instead of random sparsification of task vectors (ablation) | Task Arithmetic · 8 vision tasks (CLIP) | avg normalized accuracy: 74.00 -> 83.35 | Improved |
| use Breadcrumbs instead of Fisher Merging | multi-task merging · 8 vision tasks (CLIP) | avg normalized accuracy: 75.11 -> 83.35 | Improved |
| merge Breadcrumbs from related-task public models into a fine-tuned model | T5-base · GLUE MRPC | accuracy: 87.9 -> 90.0 | Improved |
| merge Breadcrumbs of new tasks into a fine-tuned model | T5-base · GLUE SST-2 | accuracy: 93.3 -> 94.4 | Improved |
| merge Task Arithmetic vectors into a fine-tuned model | T5-base · GLUE SST-2 | accuracy: 93.3 -> 93.5 | No clear effect |
| mask both small and large magnitude weights rather than only one tail (ablation) | Model Breadcrumbs · 8 vision tasks (ViT-B-32) | avg normalized accuracy | Improved |
| reuse hyperparameters tuned on few tasks when adding more tasks (Breadcrumbs vs Task Arithmetic) | Model Breadcrumbs · ImageNet 200-task sequence (ViT-L-14) | avg normalized accuracy | Improved |
| scale up CLIP model from ViT-B-32 to ViT-L-14 | Model Breadcrumbs · 8 vision tasks (CLIP) | avg normalized accuracy: 91.48% of fine-tuned (ViT-L-14, 8 tasks) | Improved |
| use Breadcrumbs instead of TIES | multi-task merging · 8 vision tasks (ViT-B-32) | avg normalized accuracy | Improved |
Closest papers by meaning
- Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning (ICLR 2024 · similarity 0.88)
- AdaMerging: Adaptive Model Merging for Multi-Task Learning (ICLR 2024 · similarity 0.86)
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.86)
- Parameter-Efficient Multi-Task Model Fusion with Partial Linearization (ICLR 2024 · similarity 0.85)
- No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces (ICML 2025 · similarity 0.85)
- CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
