Research map / Optimization / Distributed and compressed
Communication Compression for Decentralized Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine decentralization with 8-bit communication compression | ECD-PSGD and DCD-PSGD · ResNet-20 on CIFAR-10 | training convergence | No clear effect |
| Use decentralized communication instead of Allreduce under high latency | Decentralized_32bits/8bits · ResNet-20 on CIFAR-10, high-latency network | epoch time | Helped |
| Use 8-bit rather than full-precision communication under low bandwidth | Decentralized training · ResNet-20 on CIFAR-10, low-bandwidth network | training time | Helped |
| Use decentralized communication instead of Allreduce when bandwidth is very low | Decentralized 8-bit training · ResNet-20 on CIFAR-10, very-low-bandwidth network | epoch time | Helped |
| Scale decentralized 8-bit training from 8 to 16 nodes | ECD-PSGD and DCD-PSGD · ResNet-20 on CIFAR-10 | training convergence | No clear effect |
| Increase compression from 8 bits to 4 bits | ECD-PSGD · ResNet-20 on CIFAR-10, 16 nodes | training loss | Hurt |
| Increase compression from 8 bits to 4 bits | DCD-PSGD · ResNet-20 on CIFAR-10, 16 nodes | training convergence | Hurt |
| Replace full-precision exchanged models with naively compressed models | D-PSGD · Decentralized optimization | convergence | Hurt |
| Use difference compression instead of extrapolation compression | Compressed decentralized training · Optimization with large variation among nodes | convergence rate | Helped |
| Use difference compression instead of extrapolation compression | Compressed decentralized training · Aggressive quantization | convergence | Hurt |
| Remove the restriction on compression level by using extrapolation compression | DCD-PSGD · Aggressive quantization | convergence | Helped |
Closest papers by meaning
- Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression (NeurIPS 2022 · similarity 0.91)
- Practical Low-Rank Communication Compression in Decentralized Deep Learning (NeurIPS 2020 · similarity 0.90)
- Distributed Optimization for Overparameterized Problems: Achieving Optimal Dimension Independent Communication Complexity (NeurIPS 2022 · similarity 0.90)
- Towards Faster Decentralized Stochastic Optimization with Communication Compression (ICLR 2025 · similarity 0.88)
- Accelerated Methods with Compressed Communications for Distributed Optimization Problems Under Data Similarity (AAAI 2025 · similarity 0.87)
- Towards a Better Theoretical Understanding of Independent Subnetwork Training (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.