Research map / Large language models / Instruction and fine-tuning
Activation-Guided Consensus Merging for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ACM, a training-free model merging method that sets layer-wise task-vector coefficients from mutual information between pre-trained and fine-tuned activations, applied to long-to-short reasoning merges of Qwen and DeepSeek-R1 models.
Problem
Standard task-vector merging uses uniform coefficients across layers, ignoring functional heterogeneity; this hurts long-to-short (L2S) reasoning merges that must cut response length while keeping accuracy.
Approach
Compute per-layer mutual information between PT and FT activations on a shared calibration set (s1K, clustered and sampled), map it through a sigmoid into layer-specific coefficients that are inversely related to MI, and plug these into TA or TIES merging.
Key results
- On Qwen-7B models, ACM-TIES reaches 55.8% average accuracy (+1.3 over TIES) with a 55.3% reduction in response length versus DeepSeek-R1-7B.
- On Qwen-1.5B models, ACM-TIES reaches 43.3% average accuracy (+2.3 over TIES-Merging at 39.1%) with about 73.5% length reduction.
- On Qwen-32B, ACM-TA scores 77.4% average accuracy versus 76.9% for Task Arithmetic, with length reduction of about 19.7% versus 17.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ACM layer-wise coefficients to TIES-Merging | Qwen2.5-Math-7B L2S · Average across six math benchmarks | accuracy: 54.8 -> 55.8(+1.3) | Helped |
| Add ACM layer-wise coefficients to TIES-Merging | Qwen2.5-Math-7B L2S · Average response length across six math benchmarks | response length reduction: -53.0% -> -55.3% | Helped |
| Add ACM layer-wise coefficients to TIES-Merging | Qwen2.5-Math-1.5B L2S · Average across six math benchmarks | accuracy: 39.1 -> 43.3(+2.3) | Helped |
| Add ACM layer-wise coefficients to TA | Qwen2.5-Math-1.5B L2S · Average response length across six math benchmarks | response length reduction: -68.7% -> -73.7% | Helped |
| Add ACM layer-wise coefficients to TA | LLaMA general merging · MBPP-Pro | accuracy: 6.61 -> 6.55 (-0.06) | Hurt |
| Add ACM layer-wise coefficients to TIES | LLaMA general merging · GSM8K | accuracy: 22.90 -> 22.82 (-0.08) | Hurt |
| Add ACM layer-wise coefficients to TIES | LLaMA general merging · Across GSM8K, HellaSwag, and MBPP-Pro | accuracy: +0.08 average; GSM8K -0.08; HellaSwag +0.18; MBPP-Pro +0.12 | Mixed |
| Add ACM layer-wise coefficients to TIES | Qwen2.5-14B L2S · Average across GSM8K, MATH500, AIME24, and HumanEval-Pro | accuracy: 66.9 -> 73.8(+0.8) | Helped |
| Use 100 rather than fewer calibration pieces (ablation) | ACM-TIES on Qwen2.5-Math-1.5B · Calibration-data count | accuracy: 90.8 -> 92.2 | Helped |
| Replace 20-cluster sampling with random clustering (ablation) | ACM-TIES on Qwen2.5-Math-1.5B · Calibration-data clustering | accuracy: 91.8 -> 88.6 | Hurt |
| Use LIMO instead of s1K for calibration (ablation) | ACM-TA on Qwen2.5-Math-1.5B · GSM8K | accuracy: 76.8 -> 76.9 | No clear effect |
| Vary t within [0.5, 0.9] (ablation) | ACM · Layer-wise weight coefficients | weight coefficient stability | No clear effect |
Surprising findings
- Reported 14B and 32B results show limited length reduction for large models: accuracy gains come with only modest compression, which the authors attribute to parameter redundancy allowing both fast and slow modes to coexist.
- Cosine similarity and KL divergence were rejected as alternatives; the authors note KL is asymmetric and cosine similarity varies widely across layers.
Limitations
- Not evaluated on very large models (e.g., LLaMA-3.1-70B) or on Mixture-of-Experts models.
- Only merges models sharing the same architecture; heterogeneous merging is not addressed.
- For large models, length reduction is limited while accuracy improves; ACM's accuracy gains over RL-based length reduction are modest.
Experimental setup
Datasets: s1K LIMO GSM8K MATH500 Minerva Math OlympiadBench College Math AIME 2024 HumanEval-Pro LiveCodeBench HellaSwag MBPP-Pro
Benchmarks: GSM8K MATH500 Minerva Math OlympiadBench College Math AIME 2024 HumanEval-Pro LiveCodeBench HellaSwag MBPP-Pro IFEval
Metrics: accuracy average response length in tokens number of reflective responses
Baselines: Average Merging Task Arithmetic TIES-Merging DARE DARE-TA DARE-TIES AIM Sens-Merging DPO
Builds on: Task Arithmetic TIES-Merging DARE AIM Sens-Merging
Closest papers by meaning
- Activation-Informed Merging of Large Language Models (NeurIPS 2025 · similarity 0.91)
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.89)
- Channel Merging: Preserving Specialization for Merged Experts (AAAI 2025 · similarity 0.88)
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.88)
- Beyond Parameter Arithmetic: Sparse Complementary Fusion for Distribution-Aware Model Merging (NeurIPS 2026 · similarity 0.87)
- Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.