Research map / Architectures and efficiency / Mixture of experts
PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
Yushu Zhao, Zheng Wang, Minjia Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use element-wise dual-mask merging instead of coarse expert dropping/merging | Mixtral-8x7B at 50% sparsity · 7-task zero-shot average | average accuracy: 63.8 (HC-SMoE) -> 72.6 | Improved |
| use PuzzleMoE instead of HC-SMoE on Deepseek-MoE at 50% sparsity | Deepseek-MoE-16b · 7-task zero-shot average | average accuracy: 42.6 -> 62.1 | Improved |
| merge 3 experts per group instead of 2 (pair-wise merging) (ablation) | PuzzleMoE at 50% sparsity · WikiText-2 | perplexity: 4.36 -> 5.22 (Mixtral), 6.88 -> 7.75 (Deepseek-MoE) | Hurt |
| use evolutionary search-based pairing instead of random pairing (ablation) | PuzzleMoE at 50% sparsity · 7-task average accuracy | average accuracy: 72.6 -> 72.9 (Mixtral), 62.1 -> 62.4 (Deepseek-MoE) | No clear effect |
| calibrate on MATH instead of C4 (ablation) | PuzzleMoE on Mixtral-8x7B 50% · GSM8K and average accuracy | average accuracy: 72.6 -> 72.5; GSM8K 51.7 -> 51.7 | No clear effect |
| use math calibration data instead of C4 for the NAEE baseline | NAEE on Mixtral-8x7B 50% · GSM8K | accuracy: 28.6 -> 38.8 | Improved |
| add activation norm to saliency metric (|W|*||X||) versus weight magnitude only (ablation) | PuzzleMoE at 50% sparsity · Deepseek-MoE-16b average accuracy | average accuracy: 61.8 -> 62.1 | Improved |
| add router gate scores to the saliency metric (ablation) | PuzzleMoE at 50% sparsity · Mixtral-8x7B and Deepseek-MoE | average accuracy: 72.6 -> 72.6 (Mixtral), 62.1 -> 62.1 (Deepseek-MoE with X*gate) | No clear effect |
| vary similarity threshold tau_sim (ablation) | PuzzleMoE · WikiText-2 perplexity | perplexity | Mixed |
| shift BFloat16 exponents by 112 to free bits for packing masks and signs (ablation) | Mixtral-8x7B and Deepseek-MoE · WikiText-2 | perplexity: 4.37 -> 4.37 (Mixtral), 6.88 -> 6.88 (Deepseek-MoE) | No clear effect |
| add 3-bit group quantization on top of PuzzleMoE merging (ablation) | PuzzleMoE at 50% sparsity · Mixtral-8x7B and Deepseek-MoE | average accuracy: 72.4 at 3.35 bits vs 73.0 for AWQ on Mixtral; 61.6 vs 61.2 on Deepseek-MoE | Mixed |
| use PuzzleMoE decode-GEMM kernel with bit-packed weights instead of cuBLAS | Mixtral-8x7B 50% · end-to-end throughput | speedup: up to 1.80x end-to-end; about 2x kernel | Improved |
Later papers that compared with this method
1 later accepted paper reports PuzzleMoE 50% as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression (ICML 2026 · better on 3 of 3 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- MC#: Mixture Compressor for Mixture-of-Experts Large Models (TPAMI 2026 · similarity 0.93)
- LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · similarity 0.92)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.91)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.91)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.91)
- Toward Efficient Inference for Mixture of Experts (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
