Research map / Optimization / Optimizers
Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames multi-task model merging as a constrained optimization that minimizes the loss gap to task-specific models via data-free Taylor-approximated objective, solved by gradient descent on a modification vector projected orthogonal to a shared task subspace, with training-free norm-based merging coefficients (doGe).
Problem
Model merging of multiple fine-tuned experts into one multi-task model suffers from task conflicts and loss of task-specific and shared knowledge; existing sparsification and orthogonality methods discard information needed to match task-specific model performance.
Approach
Model merging is cast as minimizing per-task loss gaps to individual models, decomposed into a data-free objective via first-order Taylor expansion and gradient proxies from task vectors; the modification vector is optimized with gradients projected orthogonal to a shared subspace built from SVD bases of all task vectors, and merging coefficients are set as inverse task-vector norms (λ = η/||τ||).
Key results
- On ViT-B/32 8-task vision merging, doGe TA reaches 80.7% average accuracy vs 69.1% for Task Arithmetic and 80.1% for AdaMerging; doGe AM reaches 85.9%.
- On ViT-L/14, doGe AM achieves 92.6% average accuracy, close to traditional MTL at 93.5%.
- On Flan-T5-large LoRA GLUE merging, doGe TA achieves 88.0% vs 87.3% for Task Arithmetic and individual models at 89.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DOGE to Task Arithmetic | Task Arithmetic on ViT-B/32 · 8-task vision benchmark | average accuracy: 69.1 -> 80.7 | Helped |
| Apply DOGE to AdaMerging | AdaMerging on ViT-B/32 · 8-task vision benchmark | average accuracy: 80.1 -> 85.9 | Helped |
| Apply DOGE to Task Arithmetic | Task Arithmetic on Flan-T5-base (LoRA) · GLUE 8-task benchmark | average score: 77.4 -> 79.9 | Helped |
| Apply DOGE to Task Arithmetic | Task Arithmetic on Flan-T5-large (LoRA) · GLUE 8-task benchmark | average score: 87.3 -> 88.0 | Helped |
| Add Δ optimization (ablation) | DOGE TA on ViT-B/32 · 8-task vision benchmark | average accuracy: 71.9 -> 80.7 | Helped |
| Add Δ optimization (ablation) | DOGE TA on Flan-T5-base (LoRA) · GLUE 8-task benchmark | average score: 77.9 -> 79.9 | Helped |
| Constrain Δ optimization to the shared subspace (ablation) | DOGE TA on ViT-B/32 · 8-task vision benchmark | average accuracy: 77.2 -> 80.7 | Helped |
| Constrain Δ optimization to the shared subspace (ablation) | DOGE TA on Flan-T5-base (LoRA) · GLUE 8-task benchmark | average score: 79.0 -> 79.9 | Helped |
| Use task-aware λ instead of uniform λ=0.3 (ablation) | DOGE TA on ViT-B/32 · 8-task vision benchmark | average accuracy: 79.2 -> 80.7 | Helped |
| Use task-aware λ instead of uniform λ=0.3 (ablation) | DOGE TA on Flan-T5-base (LoRA) · GLUE 8-task benchmark | average score: 79.8 -> 79.9 | Helped |
| Add task-wise λ (ablation) | Task Arithmetic on ViT-B/32 · 8-task vision benchmark | average accuracy: 69.1 -> 70.7 | Helped |
| Use layer-wise λ instead of task-wise λ (ablation) | Task Arithmetic on ViT-B/32 · 8-task vision benchmark | average accuracy: 70.7 -> 71.9 | Helped |
Surprising findings
- Removing the Delta optimization drops ViT-B/32 accuracy by 8.8 points, the largest single ablation effect.
- Task vectors are inherently close to orthogonal, and further promoting orthogonality can lose shared knowledge.
Limitations
- Does not address heterogeneous model merging where task vectors have inconsistent shapes or layer counts.
- Sensitive to subspace rank choice; higher ranks add noise (optimal 10%-30%).
- Performance on SVHN drops with the shared subspace, indicating limited fit for dissimilar tasks.
Experimental setup
Datasets: SUN397 Stanford Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD GLUE (CoLA, MNLI, MRPC, QNLI, QQP, RTE, SST2, STSB)
Benchmarks: 8-task vision benchmark GLUE 8-task (Flan-T5 LoRA) unseen task generalization corruption robustness benchmarks 14/20-task merging LLM generative tasks (AlpacaEval, GSM8K, MATH, HumanEval, MBPP)
Metrics: average accuracy Spearman's rho (STSB) normalized score
Baselines: Task Arithmetic Ties-Merging Consensus Merging AWD PCB-Merging Concrete Merging AdaMerging Representation Surgery
Builds on: Task Arithmetic AdaMerging Ties-Merging AWD Concrete Merging
Closest papers by meaning
- CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging (ICML 2025 · similarity 0.91)
- AdaMerging: Adaptive Model Merging for Multi-Task Learning (ICLR 2024 · similarity 0.91)
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.91)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.91)
- Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration (NeurIPS 2025 · similarity 0.90)
- No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.