Research map / Architectures and efficiency / Mixture of experts
A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves that pruning MoE experts whose router L2-norm changed least during fine-tuning preserves test accuracy in a simplified binary setting, and empirically prunes up to 75% of experts in V-MoE and E^3-MoE with little accuracy loss.
Problem
Expert pruning in sparse mixture-of-experts models has been little studied and lacks theory; it is unclear what property identifies experts essential for a downstream task so that removing the rest preserves generalization.
Approach
Identify essential experts by the change in their router's L2 norm from pre-trained to fine-tuned weights: task-relevant experts learn a much larger router norm increase, so pruning experts with the smallest change removes task-irrelevant ones. Proven in a feature-learning framework on a simplified binary MoE, and validated on vision MoE models.
Key results
- On CIFAR-10, pruning 75% of V-MoE experts (60% of model parameters) with post-pruning fine-tuning keeps accuracy within 1% of the unpruned model.
- Without post-pruning fine-tuning, 50% of V-MoE experts (35% of the model) can be pruned on CIFAR-10 within 1% accuracy loss, excluding the penultimate MoE layer.
- Inference FLOPs and time fall roughly linearly with pruning ratio, about 40% reduction for V-MoE on CIFAR-10 at 1% tolerance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Prune experts ranked by router l2-norm change rather than randomly | V-MoE · CIFAR-10 | accuracy under pruning | Helped |
| Prune 50% of experts without post-pruning fine-tuning, while not pruning the penultimate MoE layer | V-MoE · CIFAR-10 | accuracy difference from unpruned model: 50% of the experts (35% of the whole model); within 1% | No clear effect |
| Add post-pruning fine-tuning after router norm-change pruning | V-MoE · CIFAR-10 | experts pruned while maintaining accuracy within tolerance: 75% of the experts, 60% of the whole model | Helped |
| Prune experts ranked by router l2-norm change | V-MoE · CIFAR-100 | accuracy difference from unpruned model: 50% of the experts; 41% of the whole model | No clear effect |
| Prune experts ranked by router l2-norm change | V-MoE · ImageNet | accuracy difference from unpruned model: 42% of the experts; 35% of the whole model | No clear effect |
| Prune experts ranked by router l2-norm change | E3-MoE · CIFAR-10 | accuracy difference from unpruned model: 75% of the experts; 45% of the whole model | No clear effect |
| Prune experts ranked by router l2-norm change | E3-MoE · CIFAR-100 | accuracy difference from unpruned model: 75% of the experts; 45% of the whole model | No clear effect |
| Prune experts ranked by router l2-norm change | V-MoE · CIFAR-10 | inference FLOPs: 40% | Helped |
| Prune experts ranked by router l2-norm change | E3-MoE · CIFAR-100 | inference FLOPs and time: 15% reduction of FLOPs and time | Helped |
| Prune experts ranked by router l2-norm change | V-MoE · ImageNet | inference FLOPs and time: 14% reduction of FLOPs and time | Helped |
| Prune experts using router norm change instead of importance score, absolute router magnitude, average neuron magnitude, or average change in neuron magnitude | V-MoE · CIFAR-10, CIFAR-100, and ImageNet | accuracy at acceptable model pruning ratios: within 1% | Helped |
Surprising findings
- Pruning the penultimate MoE layer without post-pruning fine-tuning is infeasible due to its sensitivity.
- The maximum allowable pruning ratio decreases as the downstream task grows in class count (CIFAR-10 > CIFAR-100 > ImageNet).
Limitations
- Theory is limited to binary classification on a simplified MoE with a fixed head, expert-choice routing, and one-layer analysis.
- Empirical results are only on vision MoE models and image classification benchmarks.
- Post-pruning fine-tuning is needed for high pruning ratios and the ImageNet setting required batch-size and learning-rate adjustments.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet
Benchmarks: CIFAR-10 CIFAR-100 ImageNet
Metrics: test accuracy model pruning ratio FLOPs inference time
Baselines: Importance score Router magnitude Average neuron magnitude Average change of neuron magnitude Random pruning
Builds on: Vision MoE Efficient ensemble of experts Expert-choice routing Feature learning framework for neural network analysis
Closest papers by meaning
- Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations (NeurIPS 2025 · similarity 0.92)
- From Sparse to Soft Mixtures of Experts (ICLR 2024 · similarity 0.91)
- DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Models (CVPR 2025 · similarity 0.91)
- Dense Backpropagation Improves Training for Sparse Mixture-of-Experts (NeurIPS 2025 · similarity 0.90)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.90)
- DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.