Research map / Architectures and efficiency / RNNs and normalization
Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives scaling laws for DiLoCo (loss and optimal inner LR, batch size, outer LR as functions of model size N and replica count M) and validates them by training 4B and 10B models, finding DiLoCo beats data-parallel training at larger scale.
Problem
Data-parallel LLM training is bottlenecked by frequent synchronization, and it is unclear how DiLoCo's behavior, including its extra outer hyperparameters, scales with model size and replica count compared with data-parallel training.
Approach
Fit scaling laws for DiLoCo from the ground up (not as a modification of data-parallel laws) as functions of model size N and number of replicas M, including optimal inner learning rate, global batch size, and outer learning rate, then use them to extrapolate and validate at 4B and 10B.
Key results
- DiLoCo with M=1 attains lower C4 evaluation loss than data-parallel at all 7 scales tested (35M to 2.4B), e.g. 2.317 vs 2.326 at 2.4B.
- At 4B and 10B with jointly-fit predicted hyperparameters, DiLoCo M=1 reaches 2.219 vs 2.224 (4B) and 2.086 vs 2.090 (10B) for data-parallel.
- Joint parametric form A*N^(alpha+beta*M)+C gives lowest held-out average residual (0.0025) vs 0.0044 for joint power law.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiLoCo with one replica instead of Data-Parallel | C4 pretraining across model scales · C4 validation, 35M–10B | evaluation loss: 3.485 -> 3.482; 2.326 -> 2.317; 2.224 -> 2.219; 2.090 -> 2.086 | Helped |
| Increase DiLoCo replicas from M=1 to M=8 | DiLoCo · C4 validation, 35M–2.4B | evaluation loss: 3.482 (-0.1%) -> 3.621 (+3.9%); 2.317 (-0.4%) -> 2.351 (+1.1%) | Hurt |
| Increase model size | DiLoCo compared with Data-Parallel · C4 validation, 35M–2.4B | relative evaluation loss: +3.9% -> +1.1% for M=8; +0.7% -> -0.1% for M=2 | Helped |
| Use DiLoCo with M=2 instead of Data-Parallel | 4B and 10B model training with predicted hyperparameters · C4 validation, 4B and 10B | evaluation loss: 2.224 -> 2.220; 2.090 -> 2.086 | Helped |
| Increase global batch size | DiLoCo rather than Data-Parallel · C4 validation and HellaSwag, varying model and batch sizes | performance stability | Helped |
| Use DiLoCo instead of Data-Parallel at larger global batch sizes | training with multiple replicas · C4 validation and zero-shot tasks | optimal global batch size | Helped |
| Increase DiLoCo replica count M | DiLoCo · C4 pretraining | optimal global batch size | Helped |
| Increase synchronization cadence H (ablation) | DiLoCo · 10B Chinchilla simulated network | bandwidth for 80% compute utilization (Gbit/s): 184.2 -> 2.0 | Helped |
| Synchronize every step (H=1) instead of less frequently (ablation) | DiLoCo · C4 validation across model sizes | evaluation loss | Hurt |
| Increase synchronization cadence H beyond 1 (ablation) | DiLoCo · C4 validation at fixed model size and replica count | evaluation loss | Hurt |
| Use DiLoCo M=1 instead of Data-Parallel under overtraining (ablation) | training on Dolma with overtraining multipliers 1, 4, and 16 · C4 validation | evaluation loss | Helped |
| Use joint rather than independent parametric fit for loss scaling (ablation) | DiLoCo loss prediction at held-out 2.4B · 2.4B held-out model scale | average residual: 0.0044 -> 0.0025 | Helped |
Surprising findings
- DiLoCo with a single replica (M=1), which gives no communication reduction, outperforms data-parallel training in evaluation loss at all model sizes tested.
- The optimal outer learning rate is essentially independent of model size and increases with synchronization cadence H, counter to the expectation that more divergence calls for a more conservative rate.
Limitations
- Scaling laws fit on models up to 2.4B; 4B and 10B results are extrapolations with hyperparameters not extensively tuned.
- Wall-clock results are idealized simulations rather than measured on real cross-datacenter networks.
- Fixed H=30 for scaling-law experiments; no theoretical grounding for the parametric forms.
Experimental setup
Datasets: C4 Dolma
Benchmarks: C4 validation loss HellaSwag PIQA ARC-Easy
Metrics: evaluation loss zero-shot accuracy residual of log-prediction compute utilization idealized wall-clock time
Baselines: Data-Parallel training Local SGD FedAvg
Builds on: DiLoCo Chinchilla scaling laws Local SGD Lookahead optimizer FedOpt
Closest papers by meaning
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.89)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.89)
- Rethinking Memory and Communication Costs for Efficient Data Parallel Training of Large Language Models (NeurIPS 2024 · similarity 0.89)
- Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.88)
- MuLoCo: Muon is a Practical Inner Optimizer for DiLoCo (ICML 2026 · similarity 0.88)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.