AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Factored Gossip DiLoCo: Reducing Blocking Communication within DiLoCo

Chamin P Hewa Koneputugodage, Thalaiyasingam Ajanthan, Sameera Ramasinghe, Hadi Mohaghegh Dolatabadi, Shamane Siriwardhana, Gil Avraham, Violetta Shevchenko, Karol Pajak, James Snewin, Alexander Long

ICML 2026 · Architectures and efficiency › RNNs and normalization · approach: Distributed LLM inference systems

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add local Mix2 (pairwise gossip) to GlobalM1 (ablation)GlobalM1 · FineWeb 1.5B, M=8, H=100validation perplexity (10B tokens): 19.42 -> 18.80Improved
use global overlapped Mix1 instead of local Mix1 (with local Mix2) (ablation)LocalM1M2 · FineWeb 1.5B, M=8, H=100validation perplexity (10B tokens): 19.37 -> 18.80Improved
replace blocking global Mix2 of DiLoCo with non-blocking global Mix1 onlyDiLoCo · FineWeb 1.5B, M=8, H=100compute utilization at 100Mbps: 36% -> 56%Improved
replace blocking global Mix2 of DiLoCo with non-blocking global Mix1 onlyDiLoCo · FineWeb 1.5B, M=8, H=100validation perplexity (10B tokens): 18.65 -> 19.42Hurt
increase inner steps H (ablation)GlobalM1 · FineWeb 1.5B, M=8max JS distance: 0.34 -> 0.42 -> 0.49Hurt
apply Mix2 only to a parameter subset (embeddings, LM head, first 2 layers) on GlobalM1 (ablation)GlobalM1 · FineWeb 1.5B, M=8, H=100max JS distance: 0.34 -> 0.22Improved
apply Mix2 only to a parameter subset instead of all parametersGlobalM1LocalM2 · FineWeb 1.5B, M=8, H=100compute utilization at 200Mbps: 66% -> 82%Improved
synchronize outer optimizer state in Mix1 (ablation)GlobalM1 · FineWeb 1.5B, M=8, H=100validation perplexity (10B tokens): 19.42 -> 20.91Hurt
add local Mix2 to GlobalM1S (ablation)GlobalM1S · FineWeb 1.5B, M=8, H=100max JS distance: 1.13 -> 0.25Improved
grow Mix2 subset from token embedding and LM head to plus layer 0 and layer 1 (ablation)GlobalM1LocalM2-subset · FineWeb 1.5Bpeak JS distance: 0.28 -> 0.26 -> 0.22Improved
add 5-step overlap from Streaming DiLoCoDiLoCo · FineWeb 1.5B, M=8, H=100compute utilization: 36% -> 36% at 100Mbps; 53% -> 54% at 200MbpsNo clear effect
use DiLoCo outer hyperparameters with NoLoCo outer stepNoLoCo · FineWeb 1.5B, H=100training stability: diverged after approximately 5k stepsHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy