Research map / Architectures and efficiency / Mixture of experts
Retraining-free Merging of Sparse MoE via Hierarchical Clustering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HC-SMoE, a retraining-free, task-agnostic expert merging method that clusters SMoE experts by average-linkage hierarchical clustering on their mean outputs over a C4 calibration set and merges them by frequency-weighted averaging.
Problem
Reducing the memory footprint of sparse mixture-of-experts LLMs without retraining and in a task-agnostic way, since existing expert pruning and merging methods are either costly, task-specific, or rely on router statistics.
Approach
Use average expert outputs over a calibration dataset as the similarity metric for expert clustering, group experts with deterministic average-linkage hierarchical clustering instead of one-shot or K-means grouping, then merge each cluster by frequency-weighted averaging of expert weights while keeping the router unchanged.
Key results
- On Qwen1.5-MoE 60->30 experts per layer (about 50% reduction), HC-SMoE (avg) reaches 0.5223 average accuracy versus 0.4528 for the best baseline F-prune.
- On Mixtral 8x7B reduced to 6 experts per layer, HC-SMoE (avg) achieves 0.6425 average accuracy against 0.6748 for the original model and 0.6363 for O-prune.
- On MedMCQA with Mixtral 6x7B, HC-SMoE reaches 0.5018 accuracy, above S-prune at 0.4794 and the original model's 0.5930 is not exceeded.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Merge Qwen 60 experts to 45 using HC-SMoE instead of the strongest pruning baseline | Qwen1.5-MoE-A2.7B-Chat · Qwen 45x2.7B, zero-shot | Average accuracy: 0.5502 -> 0.5716 | Helped |
| Merge Qwen 60 experts to 30 using HC-SMoE instead of the strongest pruning baseline | Qwen1.5-MoE-A2.7B-Chat · Qwen 30x2.7B, zero-shot | Average accuracy: 0.4528 -> 0.5223 | Helped |
| Merge Mixtral 8 experts to 6 using HC-SMoE instead of the strongest pruning baseline | Mixtral 8x7B · Mixtral 6x7B, zero-shot | Average accuracy: 0.6363 -> 0.6425 | Helped |
| Merge Mixtral 8 experts to 4 using HC-SMoE instead of O-prune | Mixtral 8x7B · Mixtral 4x7B, zero-shot | Average accuracy: 0.5728 -> 0.5877 | Helped |
| Use expert outputs rather than router-logits for hierarchical clustering (ablation) | HC-SMoE · Qwen 45x2.7B, zero-shot | Average accuracy: 0.3153 -> 0.5459 | Helped |
| Use expert outputs rather than weights for hierarchical clustering (ablation) | HC-SMoE · Qwen 45x2.7B, zero-shot | Average accuracy: 0.5234 -> 0.5459 | Helped |
| Replace hierarchical clustering with K-means using expert-output metric and random initialization (ablation) | Qwen 30x2.7B · Zero-shot | Average accuracy: 0.4518 -> 0.4988 | Hurt |
| Replace hierarchical clustering with K-means using expert-output metric and fixed initialization (ablation) | Qwen 30x2.7B · Zero-shot | Average accuracy: 0.3198 -> 0.4988 | Hurt |
| Replace frequency-weighted merging with average merging (ablation) | HC-SMoE · Qwen 30x2.7B, zero-shot | Average accuracy: 0.5223 -> 0.5240 | Helped |
| Replace hard hierarchical clustering with soft Fuzzy-Cmeans clustering (ablation) | HC-SMoE · Qwen 45x2.7B, zero-shot | Average accuracy: 0.5716 -> 0.3240 | Hurt |
| Replace hard hierarchical clustering with soft Fuzzy-Cmeans clustering (ablation) | HC-SMoE · Qwen 30x2.7B, zero-shot | Average accuracy: 0.5223 -> 0.3239 | Hurt |
| Use MATH rather than C4 for calibration (ablation) | HC-SMoE · Qwen 45x2.7B, zero-shot | Average accuracy: 0.5716 -> 0.5725 | No clear effect |
Surprising findings
- Router-logit-based similarity (M-SMoE style) collapses performance in the task-agnostic setting, e.g. Qwen 30x2.7B average drops to 0.3221.
- Merging strategy (frequency, average, fixed-dominant) has only marginal impact once clusters are good, while clustering choice matters much more.
Limitations
- Soft clustering (Fuzzy C-Means) performs very poorly because router weight merging is unclear, so the approach is limited to hard clustering.
- Uniform number of clusters per layer in main experiments; non-uniform allocation only explored as an exploratory study.
- Results are shown on zero-shot language tasks only, with the main evaluation on two MoE families.
Experimental setup
Datasets: C4 MATH CodeQA MedMCQA
Benchmarks: ARC-c ARC-e BoolQ HellaSwag MMLU OpenBookQA RTE Winogrande MedMCQA
Metrics: zero-shot accuracy average accuracy precision recall F1
Baselines: O-prune S-prune F-prune M-SMoE K-means Fuzzy C-Means ZipIt
Builds on: ZipIt O-prune S-prune M-SMoE K-means
Closest papers by meaning
- Multi-Head Mixture-of-Experts (NeurIPS 2024 · similarity 0.92)
- Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy (ICLR 2024 · similarity 0.90)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.90)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.89)
- MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.