Research map / Architectures and efficiency / RNNs and normalization
Factored Gossip DiLoCo: Reducing Blocking Communication within DiLoCo
Chamin P Hewa Koneputugodage, Thalaiyasingam Ajanthan, Sameera Ramasinghe, Hadi Mohaghegh Dolatabadi, Shamane Siriwardhana, Gil Avraham, Violetta Shevchenko, Karol Pajak, James Snewin, Alexander Long
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add local Mix2 (pairwise gossip) to GlobalM1 (ablation) | GlobalM1 · FineWeb 1.5B, M=8, H=100 | validation perplexity (10B tokens): 19.42 -> 18.80 | Improved |
| use global overlapped Mix1 instead of local Mix1 (with local Mix2) (ablation) | LocalM1M2 · FineWeb 1.5B, M=8, H=100 | validation perplexity (10B tokens): 19.37 -> 18.80 | Improved |
| replace blocking global Mix2 of DiLoCo with non-blocking global Mix1 only | DiLoCo · FineWeb 1.5B, M=8, H=100 | compute utilization at 100Mbps: 36% -> 56% | Improved |
| replace blocking global Mix2 of DiLoCo with non-blocking global Mix1 only | DiLoCo · FineWeb 1.5B, M=8, H=100 | validation perplexity (10B tokens): 18.65 -> 19.42 | Hurt |
| increase inner steps H (ablation) | GlobalM1 · FineWeb 1.5B, M=8 | max JS distance: 0.34 -> 0.42 -> 0.49 | Hurt |
| apply Mix2 only to a parameter subset (embeddings, LM head, first 2 layers) on GlobalM1 (ablation) | GlobalM1 · FineWeb 1.5B, M=8, H=100 | max JS distance: 0.34 -> 0.22 | Improved |
| apply Mix2 only to a parameter subset instead of all parameters | GlobalM1LocalM2 · FineWeb 1.5B, M=8, H=100 | compute utilization at 200Mbps: 66% -> 82% | Improved |
| synchronize outer optimizer state in Mix1 (ablation) | GlobalM1 · FineWeb 1.5B, M=8, H=100 | validation perplexity (10B tokens): 19.42 -> 20.91 | Hurt |
| add local Mix2 to GlobalM1S (ablation) | GlobalM1S · FineWeb 1.5B, M=8, H=100 | max JS distance: 1.13 -> 0.25 | Improved |
| grow Mix2 subset from token embedding and LM head to plus layer 0 and layer 1 (ablation) | GlobalM1LocalM2-subset · FineWeb 1.5B | peak JS distance: 0.28 -> 0.26 -> 0.22 | Improved |
| add 5-step overlap from Streaming DiLoCo | DiLoCo · FineWeb 1.5B, M=8, H=100 | compute utilization: 36% -> 36% at 100Mbps; 53% -> 54% at 200Mbps | No clear effect |
| use DiLoCo outer hyperparameters with NoLoCo outer step | NoLoCo · FineWeb 1.5B, H=100 | training stability: diverged after approximately 5k steps | Hurt |
Closest papers by meaning
- MuLoCo: Muon is a Practical Inner Optimizer for DiLoCo (ICML 2026 · similarity 0.85)
- Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo (NeurIPS 2025 · similarity 0.85)
- Revisiting Parameter Server in LLM Post-Training (ICLR 2026 · similarity 0.84)
- Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale (NeurIPS 2026 · similarity 0.83)
- Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUs (NeurIPS 2025 · similarity 0.83)
- ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
