Research map / Architectures and efficiency / Pruning and efficient tuning
LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LiNeS, a training-free post-training edit that scales task-vector updates linearly with layer depth, mitigating forgetting and improving multi-task, single-task, and RLHF policy merging.
Problem
Fine-tuning causes catastrophic forgetting of zero-shot generalization, and merging fine-tuned checkpoints from different tasks suffers from task interference and large performance loss.
Approach
Scale the residual (task vector) per layer with a coefficient that increases linearly with depth, so shallow layers stay close to pre-trained weights (preserving general features) while deep layers keep task-specific updates; applied post-training to single checkpoints or merged multi-task vectors.
Key results
- On 20-task ViT-L/14 benchmark, LiNeS improves Task Arithmetic by 3.1 points and Ties-Merging by 4.0 points.
- For CLIP ViT-B/32 single-task fine-tuning, LiNeS keeps 99.8% normalized target accuracy while reaching 97.9% normalized control accuracy across the 20-task benchmark.
- On the 8-task ViT-B/32 benchmark, LiNeS improves Task Arithmetic from 69.7 to 74.2 and Ties-Merging from 73.6 to 77.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply LiNeS to fine-tuned layer updates | CLIP ViT-B/32 · 8-task vision benchmark (control tasks) | accuracy: 38.0 -> 48.0 | Helped |
| Apply LiNeS to fine-tuned layer updates | CLIP ViT-B/32 · 8-task vision benchmark (target task) | accuracy: 90.5 -> 90.3 | No clear effect |
| Apply LiNeS to merged multi-task vectors | Task Arithmetic, Ties-Merging, and Consensus Merging · Vision benchmarks, 8, 14, and 20 tasks | accuracy: +4.5, +3.6, and +3.1% for 8-task ViT-B/32 | Helped |
| Apply LiNeS to merged multi-task vectors | Task Arithmetic, Ties-Merging, and Consensus Merging · 7 NLP tasks | score: +4.5, +0.4, and +1.9 | Mixed |
| Apply LiNeS to merged multi-task vectors | Task Arithmetic, Ties-Merging, and Consensus Merging · 8 QA tasks | score: +3.8, +3.0, and +0.7 | Helped |
| Apply LiNeS to merged multi-task vectors | Task Arithmetic, Ties-Merging, and Consensus Merging · 11 NLP tasks | score: +2.6, +2.4, and +0.0 | Mixed |
| Apply LiNeS to uniform and greedy model soups | CLIP ViT-B/32 · ImageNet | accuracy: +0.48% and +0.15% | Helped |
| Apply LiNeS to reward-weighted policy residuals | LLaMA-2 7B policies · Reddit Summary (rewarded soups) | reward score | Helped |
| Replace linear layer scaling with quadratic scaling (ablation) | Task Arithmetic · Vision benchmarks, 8, 14, and 20 tasks | accuracy: 74.2 -> 73.8; 69.1 -> 69.2; 63.4 -> 64.6 | Mixed |
| Apply LiNeS only to the T5 decoder rather than the encoder or both (ablation) | T5-large · 8 QA tasks (target and control) | accuracy | Mixed |
| Set both layer-scaling parameters alpha and beta above zero (ablation) | Task Arithmetic, Ties-Merging, and Consensus Merging · 8-task vision benchmark | validation accuracy | Helped |
Surprising findings
- Downscaling only the decoder of T5 preserves target accuracy and slightly improves control tasks, while encoder downscaling hurts target performance.
- Regularized fine-tuning baselines (layer-wise LR decay, freezing) do not restore control-task performance as well as post-training LiNeS.
Limitations
- Relies on heuristic alpha setting and tuning of beta; alpha heuristic is tied to merging method and number of models.
- Applied to T5 only on decoder based on empirical findings; generality of layer-depth prior across architectures is empirical.
Experimental setup
Datasets: ImageNet ImageNetSketch ImageNet-A ImageNet-R ObjectNet ImageNet-V2 Cars DTD EuroSAT GTSRB MNIST RESISC45
Benchmarks: 8-task vision benchmark 14-task vision benchmark 20-task vision benchmark 7 NLP tasks 8 QA tasks 11 NLP tasks ImageNet OOD distribution shifts Rewarded Soups summarization Pareto front
Metrics: accuracy normalized accuracy target vs control accuracy
Baselines: Task Arithmetic Ties-Merging Consensus Merging Model Soups Ada-merging aTLAS WiSE-FT Rewarded Soups
Builds on: Task Arithmetic Ties-Merging WiSE-FT Model Soups Rewarded Soups
Closest papers by meaning
- Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuning (ICLR 2025 · similarity 0.88)
- Rethinking Layer-wise Model Merging through Chain of Merges (NeurIPS 2026 · similarity 0.88)
- Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.87)
- Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration (NeurIPS 2025 · similarity 0.87)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.87)
- Contribution-Aware Structured Sparsity for Model Merging (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.