Research map / Optimization / Distributed and compressed
A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Aggregate model weights every I local updates instead of communicating every iteration (ablation) | CELGC · CIFAR-10, ResNet-56 | training loss and test accuracy by epoch | No clear effect |
| Aggregate model weights every I local updates instead of communicating every iteration (ablation) | CELGC · CIFAR-10, ResNet-56 | wall-clock training speed | Helped |
| Use I=4 local updates between communication rounds (ablation) | CELGC · Penn Treebank, AWD-LSTM | wall-clock training speed: 4x faster | Helped |
| Use I=4 local updates between communication rounds (ablation) | CELGC · Penn Treebank, AWD-LSTM | validation perplexity by epoch | No clear effect |
| Use I=4 local updates between communication rounds (ablation) | CELGC · Wikitext-2, AWD-LSTM | wall-clock training speed: 4x faster | Helped |
| Use I=4 local updates between communication rounds (ablation) | CELGC · Wikitext-2, AWD-LSTM | validation perplexity by epoch | No clear effect |
| Use I>1 local updates between communication rounds (ablation) | CELGC · ImageNet, ResNet-50 | training accuracy by epoch | No clear effect |
| Use local gradients for clipping rather than averaged gradients | CELGC · Penn Treebank and Wikitext-2 | frequency of clipping | Mixed |
| Scale the number of machines to N under the stated homogeneous-data assumptions | CELGC · Homogeneous distributed nonconvex optimization | iteration complexity: O(\Delta L_0\sigma^2/(N\epsilon^4)) | Helped |
| Skip communication and communicate after periodic local updates | CELGC · Homogeneous distributed nonconvex optimization | communication complexity: O\left(\Delta L_0\sigma\epsilon^{-3}\right) | Helped |
Closest papers by meaning
- Improved Analysis of Clipping Algorithms for Non-convex Optimization (NeurIPS 2020 · similarity 0.91)
- Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity (ICLR 2020 · similarity 0.90)
- Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees (ICML 2023 · similarity 0.89)
- Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback (NeurIPS 2019 · similarity 0.89)
- To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensions (ICLR 2025 · similarity 0.88)
- Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.