Research map / Optimization / Distributed and compressed
DES-LOC: Desynced Low Communication Adaptive Optimizers for Foundation Models
Alex Iacob, Lorenzo Sani, Mher Safaryan, Paris Giampouras, Samuel Horváth, Andrej Jovanovic, Meghdad Kurmanji, Preslav Aleksandrov, William F. Shen, Xinchi Qiu, Nicholas D. Lane
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| synchronize parameters less frequently (larger Kx) (ablation) | DES-LOC (ADOPT, 135M) · SmolLM2 mixture, 135M model | perplexity | Hurt |
| synchronize second momentum less frequently (larger Kv) (ablation) | DES-LOC (ADOPT, beta2=0.9999) · SmolLM2 mixture, 135M model | perplexity | No clear effect |
| synchronize first momentum more frequently (ablation) | DES-LOC (ADOPT, beta1=0.95) · SmolLM2 mixture, 135M model | perplexity | Mixed |
| desync momenta: Ku=3Kx and Kv=6Kx instead of Local Adam syncing all states at K | DES-LOC · SmolLM2 mixture, 135M model | communication cost at matched perplexity: 2x lower communication than Local Adam | Improved |
| sync first momentum less often than second (Ku=6Kx, Kv=3Kx) (ablation) | DES-LOC-ADOPT · 135M, 2x lower communication than Local Adam | perplexity | Hurt |
| set Ku=Kv=4Kx (ablation) | DES-LOC-ADOPT · 135M, 2x lower communication than Local Adam | perplexity | Mixed |
| split communication between momenta (Ku vs Kv) (ablation) | DES-LOC-Adam (beta1=beta2=0.95) · 135M, 2x lower communication | perplexity | No clear effect |
| sync optimizer states eventually instead of keeping them local or resetting them | DES-LOC vs FAVG+OPT · 1.7B, 40B tokens | ICL average: 50.7 -> 51.6 | Improved |
| keep optimizer states local (FedAvg + persistent states) (ablation) | FedAvg · 1.7B long horizon | training stability | Hurt |
| double worker count at step 1536 | DES-LOC and Local Adam vs heuristic baselines · 135M model | perplexity and gradient norms | No clear effect |
| use DES-LOC or Local Adam instead of DDP at billion scale | 1.7B model · ICL average | ICL average: 53.8 -> 51.6 | Hurt |
Closest papers by meaning
- MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates (ICLR 2026 · similarity 0.89)
- LoCo: Low-Bit Communication Adaptor for Large-Scale Model Training (TPAMI 2025 · similarity 0.89)
- Convergence of Distributed Adaptive Optimization with Local Updates (ICLR 2025 · similarity 0.88)
- Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam (ICLR 2023 · similarity 0.88)
- MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization (NeurIPS 2025 · similarity 0.86)
- Communication-Efficient LLM Adaptation over Decentralized GPU Meshes (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
