Research map / Large language models / Instruction and fine-tuning
Parameter Competition Balancing for Model Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PCB-Merging, a training-free, dataless model merging method that drops and rescales task-vector parameters using intra-task importance (softmax over squared magnitudes scaled by task count) and inter-task similarity scores, optionally refined by CMA-ES coefficient search.
Problem
Existing task-vector model merging methods apply uniform coefficients per task and parameter and do not balance parameter competition within and across tasks, limiting merged-model performance.
Approach
Computes a per-parameter balancing matrix combining intra-task importance (softmax of normalized squared task-vector magnitudes, scaled by number of tasks) and inter-task similarity (products of task vectors across tasks), then drops low-scoring parameters and rescales the rest before summing task vectors into the pretrained model.
Key results
- On T5-base, merging seven NLP tasks without validation reaches 73.8 average vs 69.5 for TIES-Merging (+4.3); with validation 75.4 vs 73.6.
- On ViT-B/32 eight vision tasks without validation, PCB-Merging reaches 75.9 vs 72.4 for TIES-Merging (+3.5); ViT-L/14 reaches 86.9 vs 86.0.
- On (IA)^3 eleven PEFT tasks, PCB-Merging with validation reaches 68.1 vs 66.8 for TIES-Merging (+1.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply parameter competition balancing during model merging | T5-Base · 7 NLP tasks, no validation | average performance: 69.5 -> 73.8 | Helped |
| apply parameter competition balancing during model merging | T5-Large · 7 NLP tasks, no validation | average performance: 73.6 -> 77.1 | Helped |
| apply parameter competition balancing during model merging | T0-3B IA3 adapters · 11 PEFT tasks, no validation | average performance: 64.9 -> 66.1 | Helped |
| apply parameter competition balancing during model merging | Llama-2-7B · 3 LLM tasks | average performance: 34.2 -> 35.1 | Helped |
| apply parameter competition balancing during model merging | ViT-B/32 · 8 vision tasks, no validation | average performance: 72.4 -> 75.9 | Helped |
| apply parameter competition balancing during model merging | ViT-L/14 · 8 vision tasks, no validation | average performance: 86.0 -> 86.9 | Helped |
| replace softmax with sigmoid in intra-balancing (ablation) | PCB-Merging · ViT-B/32 and T5-Base task suites | average performance: 76.3 -> 76.1; 75.4 -> 75.3 | No clear effect |
| replace softmax with ReLU in intra-balancing (ablation) | PCB-Merging · ViT-B/32 and T5-Base task suites | average performance: 76.3 -> 74.9; 75.4 -> 72.8 | Hurt |
| replace softmax with tanh in intra-balancing (ablation) | PCB-Merging · ViT-B/32 and T5-Base task suites | average performance: 76.3 -> 76.1; 75.4 -> 75.2 | No clear effect |
| replace softmax with sigmoid in inter-balancing (ablation) | PCB-Merging · ViT-B/32 and T5-Base task suites | average performance: 76.3 -> 76.2; 75.4 -> 75.3 | No clear effect |
| replace softmax with ReLU in inter-balancing (ablation) | PCB-Merging · ViT-B/32 and T5-Base task suites | average performance: 76.3 -> 76.1; 75.4 -> 75.2 | No clear effect |
| replace softmax with tanh in inter-balancing (ablation) | PCB-Merging · ViT-B/32 and T5-Base task suites | average performance: 76.3 -> 76.4; 75.4 -> 75.4 | No clear effect |
Surprising findings
- Dropping parameters can improve merged-model performance beyond individual fine-tuned models when combined with intra-balancing, not just preserve it as in DARE.
- Replacing multiplicative combination of intra- and inter-balancing with addition causes a 4.1% drop on ViT-B/32.
Limitations
- Relies on identical model architectures and shared initializations, limiting applicability.
- Does not effectively address parameter redundancy and still depends on drop operations.
- Task vector magnitudes may not always reflect parameter importance; limited theoretical understanding.
Experimental setup
Datasets: T0 mixture tasks P3 prompt templates GLUE (MRPC, RTE, CoLA, SST-2) CMMLU GSM8K HumanEval eight vision classification datasets (SUN397, Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, DTD) emotion classification domain datasets (Ober et al.)
Benchmarks: T0-style NLP tasks (paws, qasc, quartz, story_cloze, wiki_qa, winogrande, wsc) (IA)^3 eleven tasks (RTE, CB, ANLI, COPA, HellaSwag, StoryCloze, WSC, Winogrande, WiC) CMMLU GSM8K HumanEval SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST
Metrics: accuracy average normalized accuracy F1 Matthews correlation
Baselines: Averaging Task Arithmetic TIES-Merging Fisher Merging RegMean AdaMerging DARE LoraHub
Builds on: Task Arithmetic TIES-Merging DARE AdaMerging RegMean
Closest papers by meaning
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.89)
- Less is More: Efficient Model Merging with Binary Task Switch (CVPR 2025 · similarity 0.88)
- Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.87)
- Channel Merging: Preserving Specialization for Merged Experts (AAAI 2025 · similarity 0.87)
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.87)
- EMR-Merging: Tuning-Free High-Performance Model Merging (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.