Research map / Architectures and efficiency / Mixture of experts

Mixture Compressor for Mixture-of-Experts LLMs Gains More

ICLR 2025 · Architectures and efficiency › Mixture of experts · approach: Efficient MoE language model systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MC, a training-free compression framework for MoE LLMs combining expert-level mixed-precision quantization (PMQ, via linear/integer programming over expert significance) with online token-protected dynamic expert pruning (ODP).

Problem

MoE LLMs have large expert parameter memory footprints and loading latency, and activated experts are often redundant since many tokens need only one expert; prior compression handles only static quantization or dynamic pruning, and uniform low-bit quantization or routing-score pruning degrades badl

Approach

Allocate per-expert bit-widths (1/2/3-bit) by minimizing a combined significance metric (activation frequency and routing weight) times expert-wise output reconstruction error under an average bit-width constraint, then at inference skip the second expert for low-confidence tokens while protecting the ~2% most important tokens (by attention and activation magnitude) from pruning.

Key results

• At 2.54 bits, PMQ on Mixtral 8x7b reaches 67.50% average over eight zero-shot tasks vs 71.29% for 16-bit (3.8% drop) and 49.07% for BSP at 2.54 bits.
• Adding ODP at 2.54 bits keeps 66.94% average accuracy while reducing activated parameters to 3.96 GB (vs 4.53 GB) with 1.71x speedup.
• The 2.54-bit compressed Mixtral 8x7b (16.24 GB) outperforms 16-bit LLaMA2-13b (65.19%) on LM-Eval average; at 1.57 bits PMQ gets 54.49% vs 45.91% for Hessian-based allocation.

Measured effects reported in the paper

Surprising findings

• Uniform 2-bit GPTQ collapses on Mixtral 8x7b (42.67% avg, 28.6% drop), while mixed 2.54-bit allocation nearly matches 16-bit.
• Protecting only 2% of tokens during dynamic expert pruning recovers most of the performance lost by naive weight-only pruning (PPL 6.46 to 6.24 in the ablation).

Limitations

• Accuracy drops substantially on harder reasoning tasks (GSM8K, HumanEval) at 2-bit and below, e.g., GSM8K falls to 19.97 at 2.05 bits vs 58.30 for 16-bit.
• Evaluated only on Mixtral 8x7b and 8x22b; generality to other MoE architectures is not shown in the text.
• Hyperparameters alpha, beta, gamma and the pruning threshold require calibration and are set by ablation.

Experimental setup

Datasets: C4 WikiText2 Math (calibration/analysis)

Benchmarks: PIQA ARC-e ARC-c BoolQ HellaSwag Winogrande MathQA MMLU WikiText2 GSM8K HumanEval NIAH

Metrics: accuracy perplexity average accuracy loss activated parameter size speedup

Baselines: GPTQ uniform 2-bit GPTQ uniform 3-bit BSP (block score predictor) Hessian-based bit allocation LLaMA2-13b (16-bit)

Builds on: GPTQ HAWQ V2 OmniQuant HQQ Wanda

Closest papers by meaning

• Delta Decompression for MoE-based LLMs Compression (ICML 2025 · similarity 0.94)
• MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition (ICML 2025 · similarity 0.93)
• Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.93)
• MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance (ICML 2025 · similarity 0.93)
• LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · similarity 0.91)
• FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.91)