Research map / Architectures and efficiency / Quantization
MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MxMoE, a hardware-aware mixed-precision quantization framework for MoE models that allocates bitwidth per linear block via an ILP and generates fused mixed-precision Group-GEMM kernels.
Problem
MoE models have large memory footprints and heavy computation under widespread expert activation, and existing mixed-precision schemes add system overhead that erodes wall-clock gains.
Approach
Allocate bitwidths at linear-block granularity (not expert level) by jointly optimizing quantization loss and profiled runtime under expert-activation statistics, and fuse heterogeneous-precision GEMMs into one auto-generated Group-GEMM kernel.
Key results
- At 2.25-bit weight-only, MxMoE on DeepSeekV2-Lite reaches 7.01 WikiText2 perplexity vs 8.49 for GPTQ with Hadamard (2.25-16).
- MxMoE achieves 1.6-2.7x throughput over full precision for memory-bound workloads and 3-3.4x for compute-bound workloads.
- At 5-bit weight-activation, MxMoE achieves up to 29.4% speedup over W8A8 while keeping perplexity near full precision (e.g., 6.16 vs 5.92 on DeepSeekV2-Lite).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| allocate weight-only bitwidths with MxMoE rather than GPTQ* at 2.25-16 | DeepSeekV2-Lite · WikiText2 | PPL: 8.49 -> 7.01 | Helped |
| allocate weight-only bitwidths with MxMoE rather than GPTQ* at 2.25-16 | Qwen1.5-MoE · evaluation suite | Avg. accuracy: 49.07 -> 55.39 | Helped |
| allocate weight-only bitwidths with MxMoE rather than GPTQ* at 2.25-16 | Qwen2-MoE · evaluation suite | Avg. accuracy: 58.58 -> 64.57 | Helped |
| allocate weight-only bitwidths with MxMoE rather than GPTQ* at 2.25-16 | Mixtral-8×7B · evaluation suite | Avg. accuracy: 67.56 -> 67.89 | Helped |
| allocate weight-only bitwidths with MxMoE rather than GPTQ* at 3.25-16 | DeepSeekV2-Lite · evaluation suite | Avg. accuracy: 69.32 -> 69.37 | Helped |
| allocate weight-only bitwidths with MxMoE rather than GPTQ* at 3.25-16 | Qwen1.5-MoE · evaluation suite | Avg. accuracy: 66.36 -> 66.49 | Helped |
| allocate weight-only bitwidths with MxMoE rather than GPTQ* at 3.25-16 | Qwen2-MoE · evaluation suite | Avg. accuracy: 71.57 -> 71.05 | Hurt |
| allocate weight-only bitwidths with MxMoE rather than GPTQ* at 3.25-16 | Mixtral-8×7B · evaluation suite | Avg. accuracy: 77.43 -> 77.49 | Helped |
| use MxMoE mixed 5-5 instead of uniform QuaRot 4-4 | DeepSeekV2-Lite · evaluation suite | Avg. accuracy: 59.74 -> 68.64 | Helped |
| use MxMoE mixed 5-5 instead of uniform QuaRot 4-4 | Qwen1.5-MoE · evaluation suite | Avg. accuracy: 43.47 -> 66.72 | Helped |
| use MxMoE mixed 5-5 instead of uniform QuaRot 4-4 | Qwen2-MoE · evaluation suite | Avg. accuracy: 39.54 -> 71.30 | Helped |
| use MxMoE mixed 5-5 instead of uniform QuaRot 4-4 | Mixtral-8×7B · evaluation suite | Avg. accuracy: 58.94 -> 76.92 | Helped |
Surprising findings
- QuaRot at 4-4 bits yields perplexity of 110.66 on Qwen2-MoE, showing severe collapse of uniform 4-bit weight-activation quantization on MoE models.
- Linear-block-level allocation outperforms expert-level allocation in the ablation, contrary to prior expert-level mixed-precision schemes.
Limitations
- Sensitivity statistics from layer-wise loss may be inaccurate and amplified by inter-layer dependencies, causing lower gains on Qwen2-MoE at 3.25-bit.
- Improvement on Mixtral is marginal because it has only 8 experts, limiting the mixed-precision design space.
- Evaluated only on RTX-4090 GPU and four MoE models.
Experimental setup
Datasets: WikiText2
Benchmarks: Arc-Challenge Arc-Easy HellaSwag LAMBADA-openai LAMBADA-standard PIQA WinoGrande WikiText2 perplexity
Metrics: accuracy perplexity throughput (TOPS) speedup
Baselines: GPTQ QuaRot HQQ VLLM-Marlin-MoE CUTLASS
Builds on: GPTQ QuaRot CUTLASS Marlin
Closest papers by meaning
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.90)
- MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance (ICML 2025 · similarity 0.88)
- MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems (NeurIPS 2025 · similarity 0.88)
- MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.88)
- MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition (ICML 2025 · similarity 0.87)
- Unchosen Experts Can Contribute Too: Unleashing MoE Models’ Power by Self-Contrast (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.