Research map / Architectures and efficiency / Pruning and efficient tuning
Parameter-Efficient Multi-Task Model Fusion with Partial Linearization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes L-LoRA, which linearizes only the LoRA adapter modules (first-order Taylor in trainable parameters) so that task arithmetic and other merging methods fuse PEFT task vectors better than standard LoRA while keeping near-LoRA cost.
Problem
Parameter-efficient fine-tuned models merge poorly for multi-task fusion because of representational interference between tasks; PEFT fusion lags far behind full fine-tuning fusion.
Approach
Partially linearize only the adapter (LoRA) parameters via a first-order Taylor expansion around their initialization while keeping the pre-trained backbone nonlinear, so that task vectors in the tangent space are more orthogonal and fuse better with task arithmetic and related methods, at low extra cost.
Key results
- On CLIP-ViT-B-16 vision tasks, L-LoRA with task arithmetic reaches average normalized score 0.77±0.05 vs 0.64±0.03 for LoRA with task arithmetic over all task combinations.
- On Flan-T5-Base NLP tasks, L-LoRA with LoraHub reaches 0.28±0.08 average normalized score, above LoRA's 0.23±0.07 and full fine-tuning's 0.24±0.06.
- L-LoRA trains at roughly 3.24 it/s vs 3.83 it/s for LoRA on CLIP-ViT-B-16, with similar VRAM (~9.2GB vs 9.1GB), showing small overhead.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Partially linearize LoRA modules instead of using standard LoRA | CLIP-ViT-B-16 multi-task fusion · Vision tasks, all combinations | average normalized score: Simple average: 0.58±0.18 -> 0.66±0.02; Task Arithmetic: 0.64±0.03 -> 0.77±0.05; Ties-merging: 0.63±0.03 -> 0.73±0.04; L | Helped |
| Partially linearize LoRA modules instead of using standard LoRA | Flan-T5-Base multi-task fusion · NLP tasks, all combinations | average normalized score: Simple average: 0.06±0.07 -> 0.05±0.07; Task Arithmetic: 0.21±0.07 -> 0.26±0.05; Ties-merging: 0.21±0.08 -> 0.24±0.06; L | Mixed |
| Partially linearize LoRA modules instead of using standard LoRA | CLIP and Flan-T5 task-specific models · Vision and NLP task pairs | task-vector cosine similarity | Helped |
| Partially linearize LoRA modules instead of using standard LoRA | Single-task fine-tuning · Vision and NLP downstream tasks | task-specific performance | Hurt |
Surprising findings
- L-LoRA has the lowest single-task accuracy among the fine-tuning methods yet fuses better than LoRA on joint tasks.
- On NLP tasks the advantage of L-LoRA over LoRA is small, and full fine-tuning still wins in some fusion settings.
Limitations
- Single-task performance of L-LoRA is lower than LoRA and full fine-tuning.
- Gains on NLP tasks are modest and not uniform across fusion methods.
- Evaluated only on seven tasks per domain with small models (CLIP-ViT-B-16, Flan-T5-Base).
Experimental setup
Datasets: Stanford Cars DTD EuroSAT GTSRB RESISC45 SUN397 SVHN CoLA MNLI MRPC QQP RTE
Benchmarks: Stanford Cars DTD EuroSAT GTSRB RESISC45 SUN397 SVHN CoLA MNLI MRPC QQP RTE
Metrics: top-1 accuracy Spearman's rho normalized score weight disentanglement error
Baselines: simple average task arithmetic ties-merging LoraHub LoRA fine-tuning full fine-tuning
Builds on: task arithmetic ties-merging LoraHub LoRA neural tangent kernel linearization
Closest papers by meaning
- SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning (ICML 2025 · similarity 0.89)
- MTLoRA: Low-Rank Adaptation Approach for Efficient Multi-Task Learning (CVPR 2024 · similarity 0.88)
- Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuning (ICLR 2025 · similarity 0.88)
- Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMs (ICLR 2025 · similarity 0.87)
- RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness (NeurIPS 2025 · similarity 0.87)
- LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.