Research map / Optimization / Optimizers
One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining
Philip Zmushko, Egor Petrov, Nursultan Abdullaev, Khrushchev Mikhail, Samuel Horváth
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AdamW instead of Muon under one-step delay | delayed training · FineWeb-Edu (360M) | validation loss: 0.012 vs 0.046 | Hurt |
| add Error Feedback correction | AdamW async · FineWeb-Edu (360M) | validation loss: 2.890 -> 2.640 | Improved |
| add Error Feedback correction | Muon async · FineWeb-Edu (360M) | validation loss: 2.590 -> 2.583 | Improved |
| add Error Feedback correction | Adan async · FineWeb-Edu (360M) | validation loss: 2.651 -> 2.653 | Hurt |
| add Error Feedback correction | NAdam async · FineWeb-Edu (360M) | validation loss: 2.694 -> 2.703 | Hurt |
| add synchronous start before delayed updates (ablation) | MARS async · FineWeb-Edu (360M) | validation loss: 2.897 -> 2.820 | Improved |
| add synchronous start before delayed updates (ablation) | Adan async · FineWeb-Edu (360M) | validation loss: 2.651 -> 2.656 | Hurt |
| add synchronous cooldown near end of training (ablation) | Muon async · FineWeb-Edu (135M) | validation loss: 2.856 -> 2.853 | No clear effect |
| add Taylor-based DC-ASGD delay compensation (lambda 10^4 to 10^8) (ablation) | Muon async · FineWeb-Edu (135M) | validation loss | No clear effect |
| increase momentum mu from 0.95 to 0.99 (ablation) | Muon with original PipeDream · FineWeb-Edu (135M) | validation loss | Improved |
| replace PipeDream with PipeDream-2BW constant delay | async pipeline schedule at P=16 · FineWeb-Edu (135M) | validation loss: loses more than 0.03 | Improved |
| add Error Feedback with 10B MoE | Muon async · FineWeb (10B MoE) | validation loss: 1.911 -> 1.906 | Improved |
Closest papers by meaning
- PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization (ICML 2025 · similarity 0.88)
- AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism (NeurIPS 2026 · similarity 0.87)
- Nesterov Method for Asynchronous Pipeline Parallel Optimization (ICML 2025 · similarity 0.87)
- Communication-Efficient LLM Adaptation over Decentralized GPU Meshes (NeurIPS 2026 · similarity 0.86)
- DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving (ICML 2024 · similarity 0.86)
- ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
