Research map / Large language models / Instruction and fine-tuning

Channel Merging: Preserving Specialization for Merged Experts

AAAI 2025 · Large language models › Instruction and fine-tuning · approach: Fine-tuning generalization and delta compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Channel Merging, which clusters per-output-channel delta parameters across LLM experts (K-Means, K=2 groups) and merges within groups, then looks up each expert's channels at inference, reducing parameter conflicts versus model-level merging.

Problem

Merging multiple task-specific fine-tuned LLMs into one model causes parameter conflicts and performance decline as the number of experts grows, while ensembles need all experts in GPU memory and partial merging becomes storage-inefficient.

Approach

Cluster delta parameters channel by channel across experts by cosine similarity, merge only similar parameters within each cluster via task arithmetic, store K merged groups plus a small per-expert index, and at inference reconstruct the activated expert by looking up its channels from the groups.

Key results

• DARE-CM (Channel Merging with DARE) averages 69.88% on instruction tasks vs 62.89% for model-level DARE and 67.12% unmerged baseline, and 44.01% on code vs 36.70%-level DARE baseline (reported 7.31% gain).
• DARE-CM + router reaches 38.01% average on AGIEval and 54.42% on MMLU+CMMLU, versus 52.29% for Chinese-Mistral-7B-Instruct-v0.1 on MMLU+CMMLU and comparable to model ensemble (54.43%) while using 14.3B vs 26.8B total parameters (about 53%).
• Channel-level merging outperforms layer-level and model-level merging across tasks (average 48.05 vs 47.99 vs 47.29 on the four expert categories' Chinese average).

Measured effects reported in the paper

Surprising findings

• Channel-level merging with two groups matches or nearly matches four groups (model ensemble-like) on several tasks while using far fewer parameters.
• Sign-based grouping slightly outperforms KMeans on instruction and Chinese tasks, despite KMeans being the default.

Limitations

• Requires all merged experts to be fine-tuned from the same pretrained model.
• Compared with one-size-fits-all merging, Channel Merging increases the parameter count of the merged model (K/N storage ratio).
• Evaluation covers only a small set of Mistral-7B and CodeLlama-based experts.

Experimental setup

Datasets: CommonSenseQA TriviaQA GSM8K MATH HumanEval MBPP CMMLU CEval MMLU AGIEval Dolphin MetaMathQA

Benchmarks: CommonSenseQA TriviaQA GSM8K MATH HumanEval MBPP CMMLU CEval MMLU AGIEval

Metrics: accuracy

Baselines: DARE TIES BYOM Model Ensemble + router Baseline unmerged experts Layer-level merging Model-level merging

Builds on: task arithmetic TIES merging DARE BYOM MergeKit

Closest papers by meaning

• MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.91)
• Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.91)
• Activation-Informed Merging of Large Language Models (NeurIPS 2025 · similarity 0.89)
• Mixture-Trained Merging for Unified Multi-Objective Models (NeurIPS 2026 · similarity 0.89)
• OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.89)
• AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization (CVPR 2025 · similarity 0.88)