Research map / Optimization / Distributed and compressed
Identifying and Mitigating Errors in Gradient Aggregation of Distributed Data Parallel Training
Zhenheng Tang, Junlin Huang, Zichen TANG, Xueze Kang, Yuxin Wang, Peijie Dong, Shaohuai Shi, Xiaowen Chu, Bo Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add PAFT periodic parameter synchronization under small gradient noise (sigma^2 = 0.0001 or 0.001) | noised DSGD (ResNet-18, ResNet-50, GPT-2, LLaMA-2) · CIFAR-10/100 and OpenWebText/Alpaca training | test accuracy / train loss | Improved |
| add PAFT under larger noise (sigma^2 >= 0.01) | noised DSGD · CIFAR-10/100, LLM training | test accuracy / train loss | Mixed |
| add PAFT after accidental burst noise with variance 0.1 | noised DSGD (ResNet-18) · CIFAR-10, accidental large noise | test accuracy | Improved |
| add PAFT after accidental burst noise with variance 1.0 | noised DSGD (ResNet-18) · CIFAR-10, accidental large noise | test accuracy | No clear effect |
| replace per-iteration parameter sync with PAFT-Sync every H iterations (ablation) | parameter synchronization · synchronization cost | communication overhead | Improved |
| add PAFT-Dyn (dynamic frequency + overlapped async sync) to PAFT-Sync (ablation) | PAFT-Sync · ResNet-50, 32 workers | avg iteration wall-clock time (s): 0.411 -> 0.373 | Improved |
| add PAFT-Dyn to PAFT-Sync (ablation) | PAFT-Sync · ResNet-50, 16 workers | avg iteration wall-clock time (s): 0.276 -> 0.253 | Improved |
| add PAFT over standard DSGD (overhead) | DSGD · ResNet-50, 32 workers | avg iteration wall-clock time (s): 0.333 -> 0.373 | Hurt |
| learning rate decay at late training stage | noised DSGD with PAFT · ResNet-18, accidental large noise | test accuracy | Improved |
| increase gradient noise variance sigma^2 (ablation) | noised DSGD (no PAFT) · ResNet-18 on CIFAR-10, 4 workers | test accuracy | Hurt |
| larger noise (sigma^2 of 0.1 or 1.0) with PAFT on LLMs (ablation) | PAFT · GPT-2 / LLaMA-2 training and finetuning | train loss | Hurt |
Closest papers by meaning
- A Quadratic Synchronization Rule for Distributed Deep Learning (ICLR 2024 · similarity 0.87)
- Fault Tolerant ML: Efficient Meta-Aggregation and Synchronous Training (ICML 2024 · similarity 0.86)
- Flag Aggregator: Scalable Distributed Training under Failures and Augmented Losses using Convex Optimization (ICLR 2024 · similarity 0.86)
- ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload (NeurIPS 2026 · similarity 0.86)
- AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism (NeurIPS 2026 · similarity 0.86)
- ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
