Research map / Architectures and efficiency / Mixture of experts
MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoEQuant, a plug-in post-training quantization framework for MoE LLMs that uses expert-balanced self-sampling to build calibration data and affinity-guided quantization that weights calibration tokens by gate scores.
Problem
Post-training quantization of Mixture-of-Experts LLMs degrades accuracy and generalization because calibration data leaves some experts under-sampled and ignores token-expert gating affinities during weight updates.
Approach
Generate calibration data via the LLM's own self-sampling with beam-like branch pruning guided by cumulative log-probability and an expert-balance metric (EBSS), and incorporate gating scores into the layer-wise quantization loss and Hessian (AGQ), plugged into AWQ/GPTQ.
Key results
- On DeepSeek-MoE-16B at 4-bit, MoEQuant++ (on GPTQ) raises HumanEval to 25.00 vs 22.56 for GPTQ and 40.01 avg vs 39.01 for GPTQ, with FP avg 40.86.
- At 3-bit on Mixtral-8x7B, MoEQuant++ reaches 49.75 avg vs 45.04 for GPTQ and 36.05 for AWQ.
- Decoding speedup of 1.17-2.08x and memory savings of 3.27-3.74x on A6000 GPUs for 4-bit quantized models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply MoEQuant++ instead of GPTQ | Qwen-MoE-14B · 4-bit, seven-task average | avg. accuracy: 49.00 -> 49.59 | Helped |
| apply MoEQuant++ instead of GPTQ | DeepSeek-MoE-16B · 4-bit, seven-task average | avg. accuracy: 39.01 -> 40.01 | Helped |
| apply MoEQuant++ instead of GPTQ | Mixtral-8x7B · 4-bit, seven-task average | avg. accuracy: 53.42 -> 55.58 | Helped |
| apply MoEQuant++ instead of GPTQ | Qwen-MoE-14B-Chat · HumanEval | accuracy: 15.24 -> 21.95 | Helped |
| apply MoEQuant++ instead of GPTQ | DeepSeek-MoE-16B-Chat · HumanEval | accuracy: 13.41 -> 21.95 | Helped |
| use 3-bit MoEQuant++ instead of 3-bit GPTQ | DeepSeek-MoE-16B · 3-bit, seven-task average | avg. accuracy: 35.85 -> 36.47 | Helped |
| use 3-bit MoEQuant++ instead of 3-bit GPTQ | Mixtral-8x7B · 3-bit, seven-task average | avg. accuracy: 45.03 -> 49.75 | Helped |
| add AGQ to GPTQ (ablation) | DeepSeek-MoE-16B · 4-bit, seven-task average | avg. accuracy: 39.01 -> 39.50 | Helped |
| add AGQ to GPTQ (ablation) | Qwen-MoE-14B · 4-bit, seven-task average | avg. accuracy: 49.00 -> 49.02 | No clear effect |
| add AGQ to GPTQ (ablation) | Mixtral-8x7B · 4-bit, seven-task average | avg. accuracy: 53.42 -> 54.24 | Helped |
| add EBSS to GPTQ (ablation) | Qwen-MoE-14B · 4-bit, seven-task average | avg. accuracy: 49.00 -> 49.21 | Helped |
| add EBSS to GPTQ (ablation) | DeepSeek-MoE-16B · 4-bit, seven-task average | avg. accuracy: 39.01 -> 39.87 | Helped |
Surprising findings
- GPTQ with WikiText2 calibration gets the lowest WikiText2 perplexity yet performs worse on C4 and downstream tasks, indicating overfitting to the calibration set.
Experimental setup
Datasets: WikiText2 C4 EBSS self-sampled calibration data
Benchmarks: WikiText2 C4 MMLU HumanEval GSM8k BoolQ HellaSwag OpenBookQA MathQA
Metrics: perplexity accuracy average score decoder tokens/sec memory use (GB)
Baselines: RTN AWQ GPTQ
Builds on: GPTQ AWQ QuaRot
Closest papers by meaning
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.93)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.92)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.91)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.90)
- BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts (NeurIPS 2024 · similarity 0.90)
- MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.