Research map / Architectures and efficiency / Mixture of experts
MC#: Mixture Compressor for Mixture-of-Experts Large Models
Wei Huang, Yue Liao, Yukang Chen, Jianhui Liu, Haoru Tan, Si Liu, Shiming Zhang, Shuicheng Yan, Xiaojuan Qi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PMQ mixed-precision expert bit allocation instead of uniform 2-bit (ablation) | GPTQ on Mixtral 8x7b · eight zero-shot benchmarks | average accuracy (%): 42.67 -> 63.25 | Improved |
| use PMQ instead of BSP at similar bits (ablation) | Mixtral 8x7b · eight zero-shot benchmarks | average accuracy (%): 49.07 -> 67.50 | Improved |
| use PMQ instead of Hessian allocation at 1.57-bit (ablation) | Mixtral 8x7b · eight zero-shot benchmarks | average accuracy (%): 45.91 -> 54.49 | Improved |
| use PMQ instead of Hessian allocation at 2.54-bit (ablation) | Mixtral 8x7b · eight zero-shot benchmarks | average accuracy (%): 67.18 -> 67.50 | No clear effect |
| use PMQ instead of uniform 2-bit (ablation) | DeepSeek-VL2-T · five multimodal benchmarks | average score (%): 1.41 -> 59.45 | Improved |
| use PMQ instead of Hessian at 2.58-bit (ablation) | DeepSeek-VL2-S · five multimodal benchmarks | average score (%): 61.79 -> 63.66 | Improved |
| add OTP learned pruning to PMQ (ablation) | DeepSeek-VL2-S 2.07-bit · five multimodal benchmarks | average score (%): 62.40 -> 60.92 | Hurt |
| learned OTP pruning instead of random pruning (ablation) | PMQ DeepSeek-VL2-S · five multimodal benchmarks | average score (%): 52.69 -> 60.92 | Improved |
| learned OTP instead of rule-based ODP pruning (ablation) | PMQ Mixtral 8x7b · WikiText2 | perplexity (lower better): 6.62 -> 6.45 | Improved |
| add OTP to PMQ at 2.05-bit (ablation) | Mixtral 8x7b · GSM8K | accuracy (%): 19.97 -> 20.91 | No clear effect |
| add OTP to PMQ at 2.54-bit (ablation) | Mixtral 8x7b · GSM8K | accuracy (%): 37.67 -> 36.44 | Hurt |
| use PMQ+OTP at 2.05-bit instead of PMQ alone (ablation) | Mixtral 8x7b · eight zero-shot benchmarks | average accuracy (%): 63.25 -> 62.68 | Hurt |
Closest papers by meaning
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.96)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.93)
- MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance (ICML 2025 · similarity 0.93)
- Toward Efficient Inference for Mixture of Experts (NeurIPS 2024 · similarity 0.92)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.92)
- LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
