Research map / Learning theory and algorithms / Prediction and decision losses
Decentralized Training of Foundation Models in Heterogeneous Environments
Binhang Yuan, Yongjun He, Jared Quincy Davis, Tianyi Zhang, Tri Dao, Beidi Chen, Percy Liang, Christopher Re, Ce Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the evolutionary scheduler instead of random assignment (ablation) | decentralized pipeline+data parallel system · geo-distributed cases, GPT3-1.3B | throughput speedup: up to 2.7x | Improved |
| use the tailored local search gain function instead of standard Kernighan-Lin local search (ablation) | hybrid genetic scheduler · scheduler cost comparison | estimated communication cost | Improved |
| increase batch size and number of layers per device | decentralized system vs Megatron/Deepspeed · GPT3-1.3B Case 5 | gap to data center systems | Improved |
| add tensor model parallelism to Megatron | Megatron · slow/heterogeneous cases 2-5 | throughput | Hurt |
Closest papers by meaning
- Decentralized Diffusion Models (CVPR 2025 · similarity 0.88)
- Cluster Based Heterogeneous Federated Foundation Model Adaptation and Fine-Tuning (AAAI 2025 · similarity 0.86)
- Poplar: Efficient Scaling of Distributed DNN Training on Heterogeneous GPU Clusters (AAAI 2025 · similarity 0.86)
- Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism (NeurIPS 2025 · similarity 0.86)
- Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training (NeurIPS 2026 · similarity 0.85)
- Communication-Efficient LLM Adaptation over Decentralized GPU Meshes (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
