Research map / Optimization / Distributed and compressed
HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HALoS, a hierarchical fully asynchronous local SGD framework with local parameter servers per region and a global parameter server using momentum and global model merging, with a non-convex convergence proof, evaluated via trace-driven simulation on Pythia models.
Problem
Geo-distributed LLM training suffers from slow inter-region communication and straggler effects from heterogeneous accelerators; existing local SGD (DiLoCo) is synchronous and asynchronous variants (Async-Local-SGD) still wait on slow global server communication and lack theory for momentum in async
Approach
A two-tier asynchronous hierarchy: LPSs in each region asynchronously apply worker updates with momentum, accumulate K updates before sending deltas to a GPS, and merge the returned global model into local models with weight alpha, so slow inter-region links are hidden; the paper also gives the first convergence analysis of hierarchical asynchronous optimization with momentum.
Key results
- On Pythia-410M, HALoS achieves 7.5x speedup over DiLoCo, 3.9x over DiLoCo+DynUpd, and 2.1x over Async-Local-SGD in simulated geo-distributed time-to-loss.
- Pythia-160M HALoS reaches the same validation loss with 13.2B tokens vs 16.1B for Async-Local-SGD and 12.9B for DiLoCo; HALoS matches or exceeds synchronous SGD benchmark accuracy with a 68.6x training-time speedup (simulated).
- Ablation: integrating momentum, global model merging, and local accumulation together yields 3.3x faster convergence than FedAH on Pythia-70M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HALoS instead of DiLoCo | Pythia-70M training · Pythia-70M | normalized wall-clock time-to-loss: 7.2 -> 1.0 | Helped |
| Use HALoS instead of Async-Local-SGD | Pythia-160M training · Pythia-160M | normalized wall-clock time-to-loss: 1.9 -> 1.0 | Helped |
| Add dynamic local updates to DiLoCo | DiLoCo on Pythia-160M · Pythia-160M | tokens-to-loss: 12.9B -> 11.5B | Helped |
| Add global and local momentum (ablation) | HALoS compared with FedAH · Pythia-70M | tokens-to-loss: 38.1B -> 19.3B | Helped |
| Add global model merging (ablation) | HALoS with global and local momentum · Pythia-70M | tokens-to-loss: 19.3B -> 15.6B | Helped |
| Add local-server update accumulation (ablation) | HALoS with momentum and global model merging · Pythia-70M | tokens-to-loss: 15.6B -> 12.3B | Helped |
| Use global momentum on non-i.i.d. data (ablation) | HALoS · Shakespeare (non-i.i.d.) | character-level test accuracy: 50.0% -> 46.9% | Hurt |
| Use consistent grouping, with two workers per LPS | HALoS on heterogeneous clusters · Heterogeneous clusters (2, 4, 4, 6 workers per region) | normalized time-to-loss: 1.7 -> 1.0 | Helped |
| Set local update accumulation K to 32 (ablation) | HALoS on Pythia-70M · Pythia-70M | validation loss | Helped |
| Set global model merging weight alpha to 0.25 (ablation) | HALoS on Pythia-70M · Pythia-70M | validation loss | Helped |
| Set global momentum beta_g to 0.5 (ablation) | HALoS on Pythia-70M · Pythia-70M | validation loss | Helped |
| Set local momentum beta_l to 0.9 (ablation) | HALoS on Pythia-70M · Pythia-70M | validation loss | Helped |
Surprising findings
- Naive grouping of heterogeneous workers into one LPS per region causes HALoS training to diverge, requiring consistent grouping.
- Global momentum reduced accuracy by 3.1 points on non-i.i.d. Shakespeare, so the method performed better without global momentum under heterogeneity.
Limitations
- Results come from a trace-driven simulator rather than a real multi-region deployment.
- Convergence analysis relies on strong assumptions (bounded gradients, bounded staleness, bounded heterogeneity) and i.i.d. data, which is not used in practice.
- Global momentum can hurt under highly heterogeneous data; hyperparameters are tuned on Pythia-70M and transferred to other models.
Experimental setup
Datasets: The Pile (deduplicated) Shakespeare (non-i.i.d. split)
Benchmarks: Pile validation loss MMLU Hellaswag LAMBADA WinoGrande SciQ PIQA ARC-Challenge ARC-Easy Shakespeare next-character accuracy
Metrics: wall-clock time to target validation loss tokens to target validation loss test accuracy relative training speedup
Baselines: DiLoCo DiLoCo + DynUpd Async-Local-SGD FedAH Synchronous SGD (DP) DP+PP Hetero-Aware DP+PP
Builds on: DiLoCo Async-Local-SGD FedAH delayed Nesterov momentum
Closest papers by meaning
- EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models (ICLR 2025 · similarity 0.92)
- Rethinking Memory and Communication Costs for Efficient Data Parallel Training of Large Language Models (NeurIPS 2024 · similarity 0.88)
- Learn To be Efficient: Build Structured Sparsity in Large Language Models (NeurIPS 2024 · similarity 0.87)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.87)
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.87)
- Revisiting Parameter Server in LLM Post-Training (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.