Research map / Architectures and efficiency / Mixture of experts
Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M-SMoE, a routing-guided expert merging method (permutation alignment, router-logit grouping, frequency-weighted averaging), and MC-SMoE, which further compresses merged experts via low-rank plus structured sparse decomposition, cutting SMoE memory by up to 80%.
Problem
Sparsely activated Mixture-of-Experts models have high memory usage and redundant experts due to representation collapse, and naive model merging methods fail when applied to merging experts within SMoE layers.
Approach
Uses routing statistics to guide expert merging: experts are permutation-aligned, grouped around globally dominant experts by router-logit similarity, and merged with activation-frequency weights; merged experts have lower stable rank, enabling subsequent low-rank plus sparse compression.
Key results
- M-SMoE on switch-base-32 reaches 733M parameters (vs 2.0B full SMoE) with comparable or better scores, e.g., SQuAD 65.66/85.49 vs 65.39/85.81 for full SMoE and MRPC 90.69 vs 90.20.
- MC-SMoE reduces model size to 381M (about 80% memory saving) and TFLOPs from 4.65 to 3.83, with COPA 67.00 vs 68.00 for full SMoE.
- On fairseq-moe-15b zero-shot, M-SMoE (552M) beats pruning on MRPC (52.69 vs 52.20) and WinoGrande (50.43 vs 48.46), while MC-SMoE reaches 166M parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add M-SMoE routing-guided expert merging | switch-base-32 · MRPC | accuracy: 90.20 -> 90.69 | Helped |
| compress experts after M-SMoE merging | switch-base-32 · COPA | accuracy: 68.00 -> 67.00 | Hurt |
| use an adaptive instead of uniform merging ratio (ablation) | M-SMoE · COPA | accuracy: 63.00 -> 68.00 | Helped |
| use router-logits similarity instead of alternative grouping similarities (ablation) | M-SMoE · COPA | accuracy: 65.00 -> 68.00 | Helped |
| add knowledge distillation (ablation) | M-SMoE · COPA | accuracy: 64.00 -> 68.00 | Helped |
| add expert permutation alignment before merging (ablation) | M-SMoE · COPA | accuracy: 66.00 -> 68.00 | Helped |
| compress without first merging experts (ablation) | switch-base-32 · COPA | accuracy: 68.00 -> 64.00 | Hurt |
| use frequency-weighted rather than uniform expert averaging (ablation) | M-SMoE · COPA | accuracy: 64.00 -> 68.00 | Helped |
| use frequency-weighted rather than Fisher-weighted expert averaging (ablation) | M-SMoE · COPA | accuracy: 65.00 -> 68.00 | Helped |
| use a cubic rather than linear pruning-ratio schedule (ablation) | MC-SMoE · COPA | accuracy: 59.00 -> 67.00 | Helped |
| use a cubic rather than quadratic pruning-ratio schedule (ablation) | MC-SMoE · COPA | accuracy: 61.00 -> 67.00 | Helped |
Surprising findings
- Uniform averaging and Fisher-weighted merging underperform frequency-weighted merging, with Fisher weighting hurting several tasks since experts are already pretrained.
- Merged SMoE models did not reduce latency in the vanilla implementation; MC-SMoE throughput was lower than M-SMoE without specialized sparse kernels.
Limitations
- Latency is not improved in the vanilla implementation because the router is not pruned; MC-SMoE throughput drops without sparse matrix support.
- Evaluated only on two model families (Switch Transformer and fairseq MoE) and on downstream tasks; multimodal extension left to future work.
- Zero-shot results on the decoder-only 15B model show MC-SMoE sometimes underperforms M-SMoE (e.g., MRPC 47.55 vs 52.69).
Experimental setup
Datasets: SST-2 MRPC MultiRC COPA WinoGrande SQuAD v1.1 WikiQA HotpotQA OpenBookQA
Benchmarks: SST-2 MRPC MultiRC COPA WinoGrande SQuAD v1.1 WikiQA HotpotQA OpenBookQA
Metrics: accuracy exact-match F1 model size TFLOPs throughput latency
Baselines: Pruning Task-Specific pruning Averaging ZipIt REPAIR Git Re-basin Fisher-weighted merging C-SMoE
Builds on: Git Re-basin ZipIt REPAIR OPTIN/task-specific expert pruning Switch Transformers
Closest papers by meaning
- Retraining-free Merging of Sparse MoE via Hierarchical Clustering (ICML 2025 · similarity 0.90)
- Multi-Head Mixture-of-Experts (NeurIPS 2024 · similarity 0.88)
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.88)
- Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion (NeurIPS 2024 · similarity 0.86)
- Dense Backpropagation Improves Training for Sparse Mixture-of-Experts (NeurIPS 2025 · similarity 0.86)
- DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.