Research map / Architectures and efficiency / Mixture of experts
Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use asynchronous DMoE instead of model-parallel training | Learning@home · Distributed block-model throughput under simulated network latency | training throughput | Helped |
| Increase latency | Feed-forward and DMoE models · MNIST | convergence performance | Hurt |
| Use DMoE rather than a traditional feed-forward network under high latency | MNIST models · MNIST | convergence performance | Helped |
| Add a 10% expert failure rate | DMoE Transformer · WikiText-2 | language-model performance | Helped |
| Remove gradient checkpointing (ablation) | Learning@home Transformer training · Transformer blocks, 100ms latency | training throughput: approximately 9 times less throughput | Hurt |
Closest papers by meaning
- From Promise to Practice: Realizing High-performance Decentralized Training (ICLR 2025 · similarity 0.87)
- Towards a Better Theoretical Understanding of Independent Subnetwork Training (ICML 2024 · similarity 0.85)
- Decentralized Diffusion Models (CVPR 2025 · similarity 0.83)
- Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization (NeurIPS 2025 · similarity 0.83)
- Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training (ICML 2024 · similarity 0.83)
- An Empirical Study of Distributed Deep Learning Training on Edge (Student Abstract) (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.