Research map / Optimization / Distributed and compressed
MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates
Alex Iacob, Andrej Jovanovic, Mher Safaryan, Meghdad Kurmanji, Lorenzo Sani, Samuel Horváth, William F. Shen, Xinchi Qiu, Nicholas D. Lane
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase momentum decay beta1 from 0.99 to 0.995 (ablation) | MT-DAO · 16M, perplexity vs sync period Kx | perplexity increase over Kx=16: +6.2% -> +3.7% at Kx=1024 | Improved |
| use MT-DAO (slow momentum plus gradient) instead of single-momentum Local ADOPT | Local ADOPT, 4 workers, K=32 · SmolLM2 held-out val perplexity | validation perplexity: closes gap to DDP; wall clock reduced 6-27% | Improved |
| MT-DAO versus single-momentum ADOPT-DDP at 720M | 720M GPT · SmolLM2 val, 720M | time and tokens to target perplexity: 24% fewer tokens, 35% less time | Improved |
| MT-DAO versus QHADOPT-DDP across model scales | MT-DAO · SmolLM2 val, 16M/125M/720M | validation perplexity: trails at 16M, matches at 125M, about 8% faster wall-clock at 720M | Mixed |
| use outer Nesterov momentum with Local ADOPT | Local ADOPT · SmolLM2 val perplexity | validation perplexity | Improved |
| add slow momentum (beta1=0.999, omega1=0.98) to Local ADOPT (ablation) | Local ADOPT · worker update alignment | cosine similarity: > 0.95 across all four metrics | Improved |
| high-beta momenta with higher weights allow larger learning rates (ablation) | MT-DAO-ADOPT · 16M LR tuning | tolerable learning rate | Improved |
Closest papers by meaning
- AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism (NeurIPS 2026 · similarity 0.88)
- ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training (NeurIPS 2025 · similarity 0.88)
- MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization (NeurIPS 2025 · similarity 0.87)
- Convergence of Distributed Adaptive Optimization with Local Updates (ICLR 2025 · similarity 0.86)
- Rethinking Memory and Communication Costs for Efficient Data Parallel Training of Large Language Models (NeurIPS 2024 · similarity 0.86)
- Revisiting Parameter Server in LLM Post-Training (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
