AI Paper Atlas

Research map / Optimization / Distributed and compressed

Identifying and Mitigating Errors in Gradient Aggregation of Distributed Data Parallel Training

Zhenheng Tang, Junlin Huang, Zichen TANG, Xueze Kang, Yuxin Wang, Peijie Dong, Shaohuai Shi, Xiaowen Chu, Bo Li

ICML 2026 · Optimization › Distributed and compressed · approach: Local and asynchronous SGD

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add PAFT periodic parameter synchronization under small gradient noise (sigma^2 = 0.0001 or 0.001)noised DSGD (ResNet-18, ResNet-50, GPT-2, LLaMA-2) · CIFAR-10/100 and OpenWebText/Alpaca trainingtest accuracy / train lossImproved
add PAFT under larger noise (sigma^2 >= 0.01)noised DSGD · CIFAR-10/100, LLM trainingtest accuracy / train lossMixed
add PAFT after accidental burst noise with variance 0.1noised DSGD (ResNet-18) · CIFAR-10, accidental large noisetest accuracyImproved
add PAFT after accidental burst noise with variance 1.0noised DSGD (ResNet-18) · CIFAR-10, accidental large noisetest accuracyNo clear effect
replace per-iteration parameter sync with PAFT-Sync every H iterations (ablation)parameter synchronization · synchronization costcommunication overheadImproved
add PAFT-Dyn (dynamic frequency + overlapped async sync) to PAFT-Sync (ablation)PAFT-Sync · ResNet-50, 32 workersavg iteration wall-clock time (s): 0.411 -> 0.373Improved
add PAFT-Dyn to PAFT-Sync (ablation)PAFT-Sync · ResNet-50, 16 workersavg iteration wall-clock time (s): 0.276 -> 0.253Improved
add PAFT over standard DSGD (overhead)DSGD · ResNet-50, 32 workersavg iteration wall-clock time (s): 0.333 -> 0.373Hurt
learning rate decay at late training stagenoised DSGD with PAFT · ResNet-18, accidental large noisetest accuracyImproved
increase gradient noise variance sigma^2 (ablation)noised DSGD (no PAFT) · ResNet-18 on CIFAR-10, 4 workerstest accuracyHurt
larger noise (sigma^2 of 0.1 or 1.0) with PAFT on LLMs (ablation)PAFT · GPT-2 / LLaMA-2 training and finetuningtrain lossHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy