Research map / Large language models / Language, safety and interpretability
Post-Hoc Merging is Not Enough: Many-Shot Model Merging with Loss-Gap Balancing
Kyungjin Im, Miru Kim, Chanin Eom, Minhae Kwon
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| switch from post-hoc to many-shot merging for Task Arithmetic (ablation) | Task Arithmetic · Llama-3.2-3B multi-task | normalized performance: 0.706 -> 0.857 | Improved |
| switch from post-hoc to many-shot merging for ConsensusTA (ablation) | ConsensusTA · Llama-3.2-3B multi-task | normalized performance: 0.942 -> 0.945 | No clear effect |
| switch from post-hoc to many-shot merging for ConsensusTA (ablation) | ConsensusTA · Llama-3.2-3B multi-task | multi-task loss: 1.83 -> 1.49 | Improved |
| switch from post-hoc to many-shot merging for TIES (ablation) | TIES · Llama-3.2-3B multi-task | normalized performance: 0.883 -> 0.938 | Improved |
| switch from post-hoc to many-shot merging for Task Arithmetic (ablation) | Task Arithmetic · Llama-3.1-8B multi-task | normalized performance: 0.704 -> 0.785 | Improved |
| switch from post-hoc to many-shot merging for ConsensusTA (ablation) | ConsensusTA · Llama-3.1-8B multi-task | normalized performance: 0.694 -> 0.898 | Improved |
| switch from post-hoc to many-shot merging for DARE (ablation) | DARE · Qwen-3-4B multi-task | normalized performance: 0.986 -> 1.154 | Improved |
| switch from post-hoc to many-shot merging for TIES (ablation) | TIES · Qwen-3-4B multi-task | normalized performance: 1.108 -> 1.087 | Hurt |
| add loss-gap weighting to many-shot ConsensusTA (METIS) | ConsensusTA many-shot · Llama-3.2-3B multi-task | normalized performance: 0.945 -> 1.015 | Improved |
| add loss-gap weighting and consensus masking in many-shot (METIS) vs best post-hoc | METIS · Llama-3.2-3B worst-performing task | normalized performance: 0.641 -> 0.872 | Improved |
| remove many-shot merging, loss-gap weighting, or consensus masking from METIS (ablation) | METIS · Llama-3.2-3B ablation | normalized performance | Improved |
| use METIS instead of other merges for knowledge retention | METIS · CoQA Llama-3.2-3B | exact match: 0.583 -> 0.670 | Improved |
Closest papers by meaning
- Localizing Task Information for Improved Model Merging and Compression (ICML 2024 · similarity 0.89)
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.89)
- Activation-Informed Merging of Large Language Models (NeurIPS 2025 · similarity 0.89)
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.88)
- Channel Merging: Preserving Specialization for Merged Experts (AAAI 2025 · similarity 0.88)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
