Research map / Large language models / Instruction and fine-tuning
Activation-Informed Merging of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Activation-Informed Merging (AIM), a post-hoc relaxation that uses base-model activation magnitudes from a calibration set to keep salient base weights close to the pre-trained model, improving several LLM merging methods.
Problem
Merging fine-tuned LLMs from a common base can degrade performance relative to the base model and is fragile to outliers and low-quality experts; existing merging methods mostly operate in weight space and ignore activation information.
Approach
Borrowing from AWQ-style activation saliency and continual-learning weight regularization, AIM scales the merged weight update per input channel by (1 - A_pre(1-omega)), where A_pre is the normalized mean activation magnitude of the base model on a task-agnostic calibration set (Pile validation), so salient base weights change less. It is a plug-in applicable to any merging method.
Key results
- Across 5 merging methods and 4 expert combinations on Llama-2-13B experts with omega=0.4, AIM improves HV Gain in 80% of merges, with an average change of about 13% on benchmarks (excluding inf).
- Up to ~40% relative improvement in HV Gain for several merges (e.g., DARE Task Arithmetic Instruction Tuned + Math: HV Gain 0.18 to 0.26, +40.71%).
- On Qwen2.5-VL-7B merging Video-R1 and CAD-Coder, AIM raises HV Gain for all four merging methods, e.g., DARE Task Arithmetic from 0.38 to 0.45 (+17.42%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply activation-informed relaxation (AIM, ω=0.4) | DARE Ties merging · Llama-2-13B, Code + Instruction Tuned + Math | HV Gain: 0.17 -> 0.24 | Helped |
| Apply activation-informed relaxation (AIM, ω=0.4) | DARE Task Arithmetic · Llama-2-13B, Instruction Tuned + Math | HumanEval: 5.49 -> 12.20 | Helped |
| Apply activation-informed relaxation | DARE Task Arithmetic · Qwen 2.5-VL-7B, Video-R1 + CADCoder | HV Gain: 0.38 -> 0.45 | Helped |
| Apply activation-informed relaxation | TIES · Qwen 2.5-VL-7B, Video-R1 + CADCoder | CAD test100: 0.61 -> 0.65 | Helped |
| Set ω to 0.0 rather than 1.0 (ablation) | DARE Task Arithmetic · Three Llama-2 experts | HV Gain: 0.1604 -> 0.2623 | Helped |
| Set ω to 0.0 rather than 1.0 (ablation) | DARE Ties · Three Llama-2 experts | HV Gain: 0.1717 -> 0.2669 | Helped |
| Set ω to 0.0 rather than 1.0 (ablation) | Task Arithmetic · Three Llama-2 experts | HV Gain: 0.1643 -> 0.2623 | Helped |
| Set ω to 0.2 rather than 1.0 (ablation) | WIDEN · Three Llama-2 experts | HV Gain: 0.2879 -> 0.3066 | Helped |
| Set ω to 0.6 rather than 1.0 (ablation) | TIES · Three Llama-2 experts | HV Gain: 0.1096 -> 0.1155 | Helped |
| Replace activation scores with gradient-based sensitivity scores (ablation) | AIM · Llama-2-13B merging benchmarks | HV Gain | No clear effect |
| Reduce calibration data to 8 blocks (ablation) | AIM on the DARE Ties three-expert merge · Pile calibration data | HV Gain | No clear effect |
Surprising findings
- Sensitivity (gradient-based) relaxation performs about the same as activation-based relaxation, so the cheaper activation version is preferred.
- In the ablation, TIES merging degrades when omega is pushed below 0.4, and WIDEN degrades at omega=0, indicating that more relaxation is not uniformly beneficial.
Limitations
- Only the pre-trained model's activations are used; expert model activations are not exploited.
- Relaxation factor omega is a single global hyperparameter; non-linear schemes are unexplored.
- Gains are inconsistent across merging methods and benchmarks, with some drops on GSM8K or IFEval.
Experimental setup
Datasets: Pile validation (calibration)
Benchmarks: HumanEval MBPP MMLU MATH GSM8K IFEval Video MMMU VSI-Bench CAD-Coder test100 CAD-Coder R400
Metrics: accuracy hypervolume gain
Baselines: DARE Task Arithmetic DARE Ties Task Arithmetic Ties Merging WIDEN
Builds on: AWQ DARE WIDEN TIES merging Task Arithmetic
Closest papers by meaning
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.92)
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.91)
- Activation-Guided Consensus Merging for Large Language Models (NeurIPS 2025 · similarity 0.91)
- Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging (NeurIPS 2025 · similarity 0.90)
- Channel Merging: Preserving Specialization for Merged Experts (AAAI 2025 · similarity 0.89)
- $\texttt{Model-GLUE}$: Democratized LLM Scaling for A Large Model Zoo in the Wild (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.