Research map / Architectures and efficiency / RNNs and normalization
Model Merging by Uncertainty-Based Gradient Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Connects weighted parameter averaging error to gradient mismatch and proposes an uncertainty-based (Hessian/Fisher-preconditioned) merging scheme that reduces this mismatch, generalizing task arithmetic, Fisher averaging and influence-function data removal.
Problem
Weighted averaging of separately fine-tuned models is often inaccurate relative to a jointly trained target model, and it is unclear why and when it works or how to improve it, especially without retraining or tuning scaling factors.
Approach
Shows the merging error equals a weighted sum of gradient-mismatch terms; a first-order Taylor expansion of the target's gradient yields a closed-form merge that adds Hessian (diagonal Fisher) preconditioners to task arithmetic. The derivation unifies existing schemes as special cases (via H0 and Ht choices) and reveals the Bayesian/Laplace MAP interpretation.
Key results
- On RoBERTa merging four sentiment tasks with IMDB, the proposed method reaches 94.5 average accuracy versus 91.8 for task arithmetic with alpha=1 (and 94.4 with tuned alpha).
- For ViT merging of 8 tasks, the method is much more robust to scaling factors and slightly improves over the best task-arithmetic alpha.
- For toxicity removal, the method lowers GPT-2 perplexity from 30.3 (TA) to 26.9 while reducing toxicity from 13.1% to 12.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Task Arithmetic (α_t=1) with uncertainty-based gradient-matching merging | RoBERTa · sentiment tasks (Avg. accuracy) | accuracy: 91.8 -> 94.5 | Helped |
| use uncertainty-based gradient-matching merging instead of tuned Task Arithmetic | RoBERTa · sentiment tasks (Avg. accuracy) | accuracy: 94.4 -> 94.5 | Helped |
| use squared-gradient Hessian approximation instead of identity Hessian (ablation) | Ours on RoBERTa · sentiment task addition (Avg. accuracy) | accuracy: 94.3 -> 94.5 | Helped |
| sum squared gradients rather than average them (ablation) | Ours on RoBERTa · sentiment tasks (Avg. accuracy) | accuracy: 94.5 -> 94.5 | No clear effect |
| use uncertainty-based gradient-matching merging instead of Task Arithmetic | RoBERTa · Yelp | accuracy: 95.6 -> 97.3 | Helped |
| use uncertainty-based gradient-matching merging instead of Task Arithmetic | RoBERTa · SST2 | accuracy: 91.6 -> 93.7 | Helped |
| use F_sum rather than F_avg (ablation) | Ours on RoBERTa · SST2 | accuracy: 94.6 -> 93.7 | Hurt |
| use uncertainty-based gradient-matching merging instead of Task Arithmetic | T5 250M · sentiment tasks (Avg. accuracy) | accuracy: 92.6 -> 94.6 | Helped |
| use uncertainty-based gradient-matching merging instead of Task Arithmetic | GPT2 117M · toxicity removal | 100·Avg. toxicity: 9.8 -> 9.6 | Helped |
| use uncertainty-based gradient-matching merging instead of Task Arithmetic | GPT2 117M · toxicity removal | PPL: 30.3 -> 26.9 | Helped |
| use uncertainty-based gradient-matching merging instead of Task Arithmetic | GPT-J 1.3B · toxicity removal | PPL: 12.7 -> 12.8 | Hurt |
| use uncertainty-based gradient-matching merging instead of Task Arithmetic | FlanT5 250M · hallucination removal | BLEU: 18.2 -> 18.2 | No clear effect |
Surprising findings
- Identity Hessians for the task models already perform on par with tuned task arithmetic, indicating scaling from gradient-mismatch reduction matters more than the exact Hessian estimate.
- Optimizing task arithmetic scaling factors on test data still cannot match the proposed method, suggesting scalar weighting is insufficient.
Limitations
- Relies on a first-order Taylor / quadratic approximation of the loss and Hessian, which is exact only for linear models.
- Hessian approximations (diagonal Fisher or identity) are crude; H0 may be underestimated for LLM data removal, requiring tuned scaling factors there.
- Experiments are small-scale (RoBERTa-base, T5-base, GPT-2/GPT-J, ViT-B-32) rather than large LLMs.
Experimental setup
Datasets: IMDB Yelp Amazon RottenTomatoes SST2 Cars DTD EuroSAT GTSRB MNIST RESISC45 SUN397
Benchmarks: IMDB Yelp RottenTomatoes SST2 Amazon Cars DTD EuroSAT GTSRB MNIST RESISC45 SUN397
Metrics: accuracy average accuracy true average accuracy toxicity rate perplexity BLEU critic score Q^2
Baselines: Averaging Task Arithmetic Fisher Averaging RegMean TIES-Merging FA1 Sparse Finetuning
Builds on: task arithmetic Fisher averaging RegMean influence functions TIES-merging
Closest papers by meaning
- AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization (CVPR 2025 · similarity 0.87)
- EMR-Merging: Tuning-Free High-Performance Model Merging (NeurIPS 2024 · similarity 0.87)
- OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.87)
- Dynamic Model Merging Made Slim (NeurIPS 2026 · similarity 0.86)
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.86)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.