Research map / Architectures and efficiency / Pruning and efficient tuning
Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes sharpness-aware fine-tuning (SAFT, using ASAM) of task-specific models so that their parameter interference during task-arithmetic-style merging is reduced, improving merged multi-task CLIP performance.
Problem
Merging independently fine-tuned task-specific models via parameter arithmetic causes parameter interference that degrades multi-task performance; existing fine-tuning schemes that reduce interference (linearized fine-tuning) hurt single-task accuracy.
Approach
Derives a fine-tuning objective that jointly minimizes single-task loss and the loss change under merge-induced parameter perturbations, shows it resembles SAM, and approximates it with adaptive SAM (ASAM) as sharpness-aware fine-tuning (SAFT) to find flat minima that are robust to merging.
Key results
- On ViT-B/32 with task arithmetic over eight tasks, SAFT-ASAM raises average absolute accuracy from 68.23 to 69.45 over SGD, and with FTLO from 75.93 to 77.49.
- With ViT-B/16 and task arithmetic, SAFT-ASAM improves average absolute accuracy from 73.40 to 76.77.
- On four GLUE tasks with Flan-T5-base and task arithmetic, SAFT-ASAM reaches 51.48 average absolute accuracy versus 46.54 for SGD, with higher normalized accuracy (75.47) than FTTS (56.21).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with SAFT-ASAM instead of SGD | CLIP ViT-B/32 merged across eight vision tasks · Task arithmetic | average absolute accuracy: 68.23 -> 69.45 | Helped |
| fine-tune with SAFT-ASAM instead of SGD | CLIP ViT-B/32 merged across eight vision tasks · Weight averaging | average absolute accuracy: 65.72 -> 66.76 | Helped |
| fine-tune with SAFT-ASAM instead of SGD | CLIP ViT-B/32 merged across eight vision tasks · TIES merging | average absolute accuracy: 74.57 -> 75.45 | Helped |
| fine-tune with SAFT-ASAM instead of SGD | CLIP ViT-B/16 merged across eight vision tasks · Task arithmetic | average absolute accuracy: 73.40 -> 76.77 | Helped |
| fine-tune with SAFT-ASAM instead of SGD | Flan-T5-base merged across four GLUE tasks · GLUE average | average absolute accuracy: 46.54 -> 51.48 | Helped |
| use SAFT-ASAM with FTTS instead of FTTS alone | CLIP ViT-B/32 merged across eight vision tasks · Task arithmetic | average absolute accuracy: 78.35 -> 79.38 | Helped |
| use SAFT-ASAM with FTLO instead of FTLO alone | CLIP ViT-B/32 merged across eight vision tasks · Task arithmetic | average absolute accuracy: 75.93 -> 77.49 | Helped |
| combine SAFT-ASAM with FTTS and use TIES instead of task arithmetic | CLIP ViT-B/32 merged across eight vision tasks · TIES merging vs task arithmetic | average absolute accuracy: 79.38 -> 73.77 | Hurt |
| increase SAFT-ASAM fine-tuning from 2000 to 8000 steps (ablation) | ViT-B/32 across eight vision tasks · Fine-tuned task-specific models | average accuracy: 90.50 -> 91.03 | Helped |
| increase SGD fine-tuning from 2000 to 8000 steps (ablation) | ViT-B/32 across eight vision tasks · Fine-tuned task-specific models | average accuracy: 90.37 -> 90.48 | No clear effect |
| use ASAM perturbations rather than SAM perturbations (ablation) | SAFT on ViT-B/32 across eight vision tasks · Fine-tuned task-specific models | average accuracy: 90.16 -> 91.29 | Helped |
| use ASAM perturbations rather than Friendly SAM perturbations (ablation) | SAFT on ViT-B/32 across eight vision tasks · Fine-tuned task-specific models | average accuracy: 90.29 -> 91.29 | Helped |
Surprising findings
- Combining SAFT-ASAM with linear fine-tuning (FTTS/FTLO) and TIES merging degrades performance relative to task arithmetic, which the authors attribute to TIES trimming important parameters.
- Random-perturbation flat-minima methods RWP and SAGM perform worse than SGD after merging, while SWA slightly improves it.
Limitations
- Relies on approximating merge-induced perturbations as random/parameter-dependent perturbations, so the connection to model merging is heuristic.
- Benefits are modest in some settings (e.g., ViT-B/16 weight averaging gains 0.26 points) and TIES combinations can hurt performance.
- Experiments are limited to CLIP image classification and four GLUE tasks with Flan-T5-base.
Experimental setup
Datasets: Cars DTD EuroSAT GTSRB MNIST RESISC45 SUN397 SVHN CoLA MRPC RTE SST-2
Benchmarks: Cars DTD EuroSAT GTSRB MNIST RESISC45 SUN397 SVHN CoLA MRPC RTE SST-2
Metrics: absolute accuracy normalized accuracy weight disentanglement error cross-task linearity loss landscape
Baselines: SGD fine-tuning FTTS FTLO weight averaging TIES-merging SWA RWP SAGM
Builds on: sharpness-aware minimization adaptive sharpness-aware minimization task arithmetic TIES-merging fine-tuning in the tangent space
Closest papers by meaning
- Parameter-Efficient Multi-Task Model Fusion with Partial Linearization (ICLR 2024 · similarity 0.88)
- LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging (ICLR 2025 · similarity 0.88)
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.88)
- EMR-Merging: Tuning-Free High-Performance Model Merging (NeurIPS 2024 · similarity 0.87)
- Parameter Competition Balancing for Model Merging (NeurIPS 2024 · similarity 0.87)
- Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.