Research map / Architectures and efficiency / Pruning and efficient tuning
Merging Models with Fisher-Weighted Averaging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fisher-weighted merging instead of isotropic averaging | intermediate-task merging · RTE target, MNLI donor | accuracy: 72.2 -> 73.2 | Helped |
| Fisher-weighted merging instead of isotropic averaging | intermediate-task merging · RTE target, QNLI donor | accuracy: 66.8 -> 66.5 | No clear effect |
| Fisher merging instead of isotropic merging for ensembling BERT-base checkpoints | parameter averaging · RTE, MRPC, SST-2 | accuracy | Helped |
| Fisher merging instead of isotropic merging in WiSE-FT robust fine-tuning | ViT-B/16 · ImageNet vs OOD | IID accuracy: about 1% higher IID accuracy | Helped |
| gradient-based intermediate-task training with MNLI donor instead of merging | BERT-base · RTE target, MNLI donor | accuracy: 73.2 -> 77.4 | Helped |
| gradient-based intermediate-task training with CoLA donor | BERT-base · RTE | accuracy: 63.7 -> 62.1 | Hurt |
| gradient-based intermediate training with RTE donor on MRPC target | BERT-base · MRPC | accuracy: 84.5 -> 82.2 | Hurt |
| Fisher merging on top of gradient-based MNLI-then-RTE model | BERT-base · RTE | accuracy | Helped |
| Fisher merging RTE model with another RTE checkpoint | RoBERTa-large · RTE | accuracy: +2.2 points | Helped |
| add second intermediate task in gradient-based sequential training | RoBERTa-large · RTE | accuracy | Hurt |
| estimate Fisher with 256 examples instead of full data (ablation) | Fisher merging · RTE target, MNLI donor | accuracy: 73.4 -> 72.7 | No clear effect |
| Fisher merging instead of fine-tuning (compute) | BERT-base · RTE | FLOPs: 6x lower FLOPs | Helped |
Closest papers by meaning
- Model Merging by Uncertainty-Based Gradient Matching (ICLR 2024 · similarity 0.88)
- Parameter Competition Balancing for Model Merging (NeurIPS 2024 · similarity 0.87)
- Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.87)
- Scalable Model Merging with Progressive Layer-wise Distillation (ICML 2025 · similarity 0.86)
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.86)
- EMR-Merging: Tuning-Free High-Performance Model Merging (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.