Research map / Large language models / Instruction and fine-tuning
Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that transformer submodules (layers, attention, MLP) are far more linear than full models and proposes a training-free, closed-form submodule-wise task arithmetic merging method that outperforms standard task arithmetic on Llama-2 merges.
Problem
Task arithmetic merging of fine-tuned LLMs performs poorly when the full model lacks the linearity needed for weight-space combination; existing fixes require retraining with original data for linearization.
Approach
Linearity already exists within submodules (layers, attention, MLPs) even when the full model is nonlinear; so merge each submodule independently with closed-form optimal weights derived from feature-space least squares on about 30 samples per task, without retraining.
Key results
- On Llama-2-13B merging math, coding and translation, Attn/MLP-level merging reaches 51.05 average vs 48.24 for Task Arithmetic and 50.14 for the fine-tuned models.
- On Llama-2-7B, the layer-level method outperforms Task Arithmetic on three-model merging (42.68 vs 39.59), but is weaker on the coding and translation pair.
- Model-level application of the closed-form weights mostly fails (e.g., 13.38 on Llama-2-7B math+translate), and head-level merging is unstable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use attention/MLP-level merging instead of Task Arithmetic | Llama-2-13B · Math & Coding | average score: 34.60 -> 35.11 | Helped |
| use attention/MLP-level merging instead of Task Arithmetic | Llama-2-13B · Math & Translate | average score: 64.72 -> 66.11 | Helped |
| use submodule merging instead of DARE | Llama-2-13B · Coding & Translate | average score: 54.33 -> 52.51 | Hurt |
| use attention/MLP-level merging instead of Fine-tuned Model | Llama-2-13B · Math, Coding & Translate | average score: 50.14 -> 51.05 | Helped |
| use attention/MLP-level merging instead of Task Arithmetic | Llama-2-7B · Math & Translate | average score: 54.65 -> 55.86 | Helped |
| use layer-level merging instead of Task Arithmetic | Llama-2-7B · Coding & Translate | average score: 51.11 -> 50.64 | Hurt |
| use attention/MLP-level merging instead of Task Arithmetic | Llama-2-7B · Math, Coding & Translate | average score: 39.59 -> 42.88 | Helped |
| use attention/MLP-level merging instead of Task Arithmetic | Qwen-2.5-7B · Math, Coding & Translate | average score: 76.90 -> 77.51 | Helped |
| decompose at layer level rather than merging at model level (ablation) | Llama-2-7B · Math & Translate | average score: 13.38{ $\pm$0.57} -> 49.79{ $\pm$0.43} | Helped |
| decompose at attention/MLP level rather than merging at model level (ablation) | Llama-2-13B · Coding & Translate | average score: 15.75{ $\pm$7.60} -> 52.51{ $\pm$0.26} | Helped |
| divide attention modules into individual heads while retaining MLP merging (ablation) | Llama-2-7B · Math & Coding | average score: 27.33{ $\pm$0.10} -> 13.08{ $\pm$9.33} | Hurt |
| divide attention modules into individual heads while retaining MLP merging (ablation) | Llama-2-13B · Math, Coding & Translate | average score: 51.05{ $\pm$0.16} -> 22.59{ $\pm$19.65} | Hurt |
Surprising findings
- Applying the same closed-form merging at the whole-model level usually fails, while submodule-level application succeeds.
- Head-level merging, despite good measured linearity, yields unstable and often collapsed performance.
Limitations
- Gains are inconsistent: Llama-2-7B coding+translate and some two-model settings fall below Task Arithmetic or TIES.
- Head-level decomposition is unstable, and model-level application does not work, so the method is only effective under linear conditions.
- Evaluated on a small set of tasks and two model families/sizes; authors note the approach has no tunable hyperparameters but its generality is limited.
Experimental setup
Datasets: GSM8K Code Alpaca zh-en translation dataset Cosmos QA ROPES Winogrande
Benchmarks: GSM8K HumanEval zh-en translation evaluation Cosmos QA ROPES Winogrande
Metrics: accuracy average accuracy across merged tasks non-linearity score projection distance cosine similarity
Baselines: Weight Average Task Arithmetic DARE TIES Breadcrumbs Consensus TA Model-level closed-form merge
Builds on: Task Arithmetic Ortiz-Jimenez et al. linearity and task arithmetic (tangent-space / projection d DARE Weight averaging
Closest papers by meaning
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.89)
- OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.87)
- Parameter-Efficient Multi-Task Model Fusion with Partial Linearization (ICLR 2024 · similarity 0.87)
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.87)
- AdaMerging: Adaptive Model Merging for Multi-Task Learning (ICLR 2024 · similarity 0.87)
- No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.