Research map / Architectures and efficiency / Mixture of experts
GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
Jianing Deng, Song Wang, Dongwei Wang, Zijie Liu, Tianlong Chen, Huanrui Yang, Jingtong Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace layer-wise local bit allocation (PMQ) with global expert-level LP allocation plus router fine-tuning | Mixtral-8x7B 1.5 bits · C4 perplexity | perplexity: 20.77 -> 16.20 | Improved |
| add global router fine-tuning after quantization (ablation) | GEMQ Mixtral-8x7B 1.5 bits · WikiText2 / C4 | perplexity: WT2 9.66 -> 7.69; C4 23.53 -> 17.18 | Improved |
| replace quantized router logits with recorded FP logits (rigid FP-matching) (ablation) | GEMQ Mixtral-8x7B 1.5 bits · WikiText2 / C4 | perplexity: WT2 9.66 -> 9.29; C4 23.53 -> 22.72 | No clear effect |
| estimate importance with a 2-bit quantized model instead of FP (progressive quantization) (ablation) | GEMQ Mixtral-8x7B 1.5 bits · WikiText2 | perplexity: 9.66 -> 9.33 | Improved |
| fine-tune routers of the 2-bit model before using it for importance estimation (ablation) | GEMQ Mixtral-8x7B 1.5 bits · WikiText2 | perplexity: 9.33 -> 9.01 | Improved |
| calibrate on C4 only vs C4 only for GSM8K | GEMQ vs PMQ Mixtral-8x7B 2.5 bits · GSM8K | accuracy (%): PMQ 33.06 vs GEMQ 31.77 | Hurt |
| add 128 MATH sequences to C4 calibration set (ablation) | GEMQ Mixtral-8x7B 2.5 bits · GSM8K | accuracy (%): 31.77 -> 42.30 | Improved |
| naive PMQ extension: apply layer-wise coefficients in a global LP (ablation) | global expert bit allocation Mixtral-8x7B · C4 perplexity | perplexity | Hurt |
| apply FIM approximation to weight Hessian instead of output Hessian (ablation) | global expert bit allocation Mixtral-8x7B · C4 perplexity | perplexity | Hurt |
| two-step allocation (layer budgets then experts) vs single global LP (ablation) | global expert bit allocation Mixtral-8x7B · C4 perplexity | perplexity | Hurt |
| increase calibration sequences beyond 128 (ablation) | GEMQ · calibration size | quantization performance | No clear effect |
| GEMQ vs SpQR at 1.5 bits | Mixtral-8x7B · 7-task zero-shot average | accuracy (%): 31.87 -> 52.00 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- MoEQuant (3-16) — MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance (ICML 2025) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance (ICML 2025 · similarity 0.92)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.91)
- MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design (ICML 2025 · similarity 0.91)
- SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models (ICML 2025 · similarity 0.90)
- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (ICML 2024 · similarity 0.90)
- OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
