AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

Yushu Zhao, Zheng Wang, Minjia Zhang

ICML 2026 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use element-wise dual-mask merging instead of coarse expert dropping/mergingMixtral-8x7B at 50% sparsity · 7-task zero-shot averageaverage accuracy: 63.8 (HC-SMoE) -> 72.6Improved
use PuzzleMoE instead of HC-SMoE on Deepseek-MoE at 50% sparsityDeepseek-MoE-16b · 7-task zero-shot averageaverage accuracy: 42.6 -> 62.1Improved
merge 3 experts per group instead of 2 (pair-wise merging) (ablation)PuzzleMoE at 50% sparsity · WikiText-2perplexity: 4.36 -> 5.22 (Mixtral), 6.88 -> 7.75 (Deepseek-MoE)Hurt
use evolutionary search-based pairing instead of random pairing (ablation)PuzzleMoE at 50% sparsity · 7-task average accuracyaverage accuracy: 72.6 -> 72.9 (Mixtral), 62.1 -> 62.4 (Deepseek-MoE)No clear effect
calibrate on MATH instead of C4 (ablation)PuzzleMoE on Mixtral-8x7B 50% · GSM8K and average accuracyaverage accuracy: 72.6 -> 72.5; GSM8K 51.7 -> 51.7No clear effect
use math calibration data instead of C4 for the NAEE baselineNAEE on Mixtral-8x7B 50% · GSM8Kaccuracy: 28.6 -> 38.8Improved
add activation norm to saliency metric (|W|*||X||) versus weight magnitude only (ablation)PuzzleMoE at 50% sparsity · Deepseek-MoE-16b average accuracyaverage accuracy: 61.8 -> 62.1Improved
add router gate scores to the saliency metric (ablation)PuzzleMoE at 50% sparsity · Mixtral-8x7B and Deepseek-MoEaverage accuracy: 72.6 -> 72.6 (Mixtral), 62.1 -> 62.1 (Deepseek-MoE with X*gate)No clear effect
vary similarity threshold tau_sim (ablation)PuzzleMoE · WikiText-2 perplexityperplexityMixed
shift BFloat16 exponents by 112 to free bits for packing masks and signs (ablation)Mixtral-8x7B and Deepseek-MoE · WikiText-2perplexity: 4.37 -> 4.37 (Mixtral), 6.88 -> 6.88 (Deepseek-MoE)No clear effect
add 3-bit group quantization on top of PuzzleMoE merging (ablation)PuzzleMoE at 50% sparsity · Mixtral-8x7B and Deepseek-MoEaverage accuracy: 72.4 at 3.35 bits vs 73.0 for AWQ on Mixtral; 61.6 vs 61.2 on Deepseek-MoEMixed
use PuzzleMoE decode-GEMM kernel with bit-packed weights instead of cuBLASMixtral-8x7B 50% · end-to-end throughputspeedup: up to 1.80x end-to-end; about 2x kernelImproved

Later papers that compared with this method

1 later accepted paper reports PuzzleMoE 50% as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy