Research map / Architectures and efficiency / Mixture of experts
LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LSH-MoE, which compresses all-to-all token communication in MoE expert-parallel training by clustering tokens with cross-polytope LSH and sending only centroids, with residual error compensation.
Problem
All-to-all communication in expert-parallel MoE training consumes a large share of training time (about 45% on average in profiled models), limiting efficiency and scalability.
Approach
Exploit token similarity in all-to-all traffic: cluster tokens online with cross-polytope locality-sensitive hashing, transmit only cluster centroids through experts, and restore per-token outputs by adding back stored residuals to the expert output of each centroid.
Key results
- LSH-MoE reaches equivalent perplexity 1.6x faster for RoBERTa-MoE and 2.2x faster for T5-MoE than the original models.
- Fine-tuning GPT-MoE (15B and 52B) on GLUE gives 1.2x-1.5x speedup with accuracy within about 0.5 points of the original on most tasks.
- On Swin-MoE fine-tuning on ImageNet-1K, LSH-MoE gives 1.28x throughput (184.3 to 236.6 samples/s) with 11.7% compression rate and Top-1 84.5% vs 84.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply LSH-MoE compression with residual compensation | GPT-MoE (15B) fine-tuning · GLUE | accuracy | No clear effect |
| Apply LSH-MoE compression with residual compensation | GPT-MoE (52B) fine-tuning · GLUE | accuracy | No clear effect |
| Apply LSH-MoE compression with residual compensation | Swin-MoE fine-tuning · ImageNet-1K | Top-1 accuracy: 84.7% -> 84.5% | No clear effect |
| Apply LSH-MoE compression with residual compensation | Swin-MoE fine-tuning · ImageNet-1K | samples/s: 184.3 -> 236.6 | Helped |
| Apply LSH-MoE with residual compensation | RoBERTa-MoE pre-training · BooksCorpus and English Wikipedia | time to equivalent perplexity: 1.6× | Helped |
| Apply LSH-MoE with residual compensation | T5-MoE pre-training · industry dataset | time to equivalent perplexity: 2.2× | Helped |
| Remove residual-based error compensation (ablation) | LSH-MoE · RoBERTa-MoE pre-training | perplexity: +0.3 | Hurt |
| Increase the number of hash functions (2, 4, 6, 8, or 10) (ablation) | LSH-MoE · GPT-MoE (15B), MNLI and SST-2 | convergence quality | Helped |
| Increase the number of hash functions (2, 4, 6, 8, or 10) (ablation) | LSH-MoE · GPT-MoE (15B), MNLI and SST-2 | compression rate | Hurt |
| Use cross-polytope hashing instead of spherical-plane hashing (ablation) | LSH-MoE · GPT-MoE (15B), MNLI and SST-2 | convergence quality | Helped |
Surprising findings
- Removing error compensation raised perplexity by 0.3 points at equal training duration, showing residual compensation matters for activation compression.
- Their scalability analysis suggests the all-to-all to compute time ratio stays roughly constant as servers and model size grow.
Limitations
- Only evaluates MoE models; the approach is not tested on other architectures.
- Compression relies on token redundancy, and the paper's gains depend on hash configuration (6 hash functions chosen empirically).
Experimental setup
Datasets: BooksCorpus English Wikipedia industry pre-training dataset GLUE ImageNet-1K
Benchmarks: GLUE ImageNet-1K
Metrics: perplexity accuracy (top-1, top-5) zero-shot accuracy training speedup compression rate samples/s
Baselines: Original MoE training (no compression) Spherical hashing (SP) TA-MoE Pre-gated MoE DeepSpeed-MoE
Builds on: Switch Transformer Reformer DeepSpeed-MoE Hash Layer
Closest papers by meaning
- Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training (ICML 2024 · similarity 0.90)
- MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.90)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.88)
- Dense Backpropagation Improves Training for Sparse Mixture-of-Experts (NeurIPS 2025 · similarity 0.88)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.88)
- Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.