Research map / Architectures and efficiency / Mixture of experts
RCMoE: A Communication-Efficient Random Compression Framework for Resource-Constrained Mixture-of-Experts Training
Donglei Wu, Xiao Cai, Jinglei Tan, Jinda Jia, Guangming Tan, Dingwen Tao, Wen Xia, Zhihong Tian
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| row-wise quantization instead of column-wise or global quantization of intermediate results (8-bit) (ablation) | MoE all-to-all compression · MoE training validation accuracy | validation accuracy | Improved |
| row-wise instead of global quantization (ablation) | MoE all-to-all compression · MoE training runtime | runtime | Improved |
| increase quantization bit-width from 6 to 8 bits (ablation) | linear quantization of intermediate results · MoE training | accuracy vs compression ratio | Mixed |
| use unbiased LSQ instead of biased global linear quantization (ShapeShifter) (ablation) | all-to-all compression · VC100 | convergence stability | Improved |
| use LSQ instead of QSGD for intermediate results (ablation) | all-to-all compression · MoE tasks | training convergence | Improved |
| use PTS instead of MSTop-K or Rand-K sparsification (ablation) | all-reduce compression · MoE tasks at same sparsity | training quality | Improved |
| use PTS instead of DGC sparsification (ablation) | all-reduce compression · ViT-based vision tasks | training quality | Improved |
| RCMoE (LSQ+PTS) vs best compression combinations | GW2 · GW2 validation perplexity | perplexity: 311.1 -> 304.5 | Improved |
| RCMoE vs no compression | VT200 · VT200 validation accuracy | accuracy: 0.5324 -> 0.5861 | Improved |
| naive compression combinations (Top-k, linear quantization) applied to MoE training | MoE training · MoE tasks | accuracy: 18.9%-66.7% accuracy degradation | Hurt |
| PWLQ/ShapeShifter+DGC combos in speedup | MoE training time · VT200 | speedup: 0.81x | Hurt |
| scale to more GPUs with RCMoE | RCMoE · 4 to 16 GPUs | speedup | Improved |
Closest papers by meaning
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.91)
- MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition (ICML 2025 · similarity 0.90)
- DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning (NeurIPS 2025 · similarity 0.89)
- LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · similarity 0.89)
- Delta Decompression for MoE-based LLMs Compression (ICML 2025 · similarity 0.89)
- FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
