AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees

Mohammed Nowaz Rabbani Chowdhury, Kaoutar El Maghraoui, Hsinyu Tsai, Naigang Wang, Geoffrey W. Burr, Liu Liu, Meng Wang

ICLR 2026 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
order experts by router l2-norm change (smaller = higher precision) instead of activation frequency or weights (ablation)Switch Transformer (CNNDM) mixed precision · CNN/DailyMailquality retention at low bits: preserved down to 2.5 avg bits vs worse baselinesImproved
add MaxVar reordering on top of router-norm ordering (ablation)Switch Transformer (CNNDM) · CNN/DailyMailquality retention at low bits: preservation extended to 2.125 bitsImproved
use router norm + MaxVar instead of PMQ above 2.0 bitsMixtral 8x7B · 8 zero-shot tasks avgaverage accuracy: 68.38 vs 67.53 at 2.5 bitsImproved
use router norm + MaxVar instead of PMQ at 2.0 and 1.75 bitsMixtral 8x7B · 8 zero-shot tasks avgaverage accuracy: 62.56 vs 62.83 at 2.0; 58.95 vs 60.53 at 1.75Hurt
expert-wise router-norm method vs layer-wise/group-wise methodsMixtral 8x7B · 8 zero-shot tasks avgaverage accuracy: 68.38 vs 49.07 (BSP) at 2.5 bits; 62.56 vs 44.49 (Slim-LLM) at 2.0Improved
set MaxVar threshold zeta to 3.0 (vs 1.0) (ablation)Router norm + MaxVar on Mixtral 8x7B · 8 tasks avg, 2.0 bitsaverage accuracy: 60.44 -> 62.56Improved
use router norm instead of PMQ for bit assignmentMixtral 8x7B · assignment overheadcompute time: 2227s and 110 GB GPU -> negligible, no GPUImproved
assign higher precision to less frequently activated expertsMixtral · Wikitext2 inference timeinference timeImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy