Research map / Learning theory and algorithms / Matrix and tensor methods
Practical Low-Rank Communication Compression in Decentralized Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use power iteration for the projection vector instead of random projections (ablation) | PowerGossip · WikiText-2 LSTM, 90 epochs | test loss: 4.627 -> 4.565 | Helped |
| increase power iterations per gradient step from 8 to 32 (ablation) | PowerGossip · WikiText-2 LSTM | test loss: 4.73 -> 4.57 | Helped |
| increase power iterations from 1 to 2 (ablation) | PowerGossip · CIFAR-10 ResNet-20 | test accuracy: 91.7% -> 91.9% | Helped |
| use PowerGossip (32 iterations) instead of uncompressed D-PSGD | decentralized SGD · WikiText-2 LSTM | test loss: 4.58 -> 4.57 | Helped |
| use PowerGossip (2 iterations) instead of uncompressed D-PSGD | decentralized SGD · CIFAR-10 ResNet-20 | test accuracy: 92.1% -> 91.9% | No clear effect |
| use PowerGossip (32 iterations) instead of Choco (top-1%) | compressed decentralized SGD · WikiText-2 LSTM | test loss: 5.04 -> 4.57 | Helped |
| use PowerGossip (2 iterations) instead of Choco (Sign+Norm) | compressed decentralized SGD · CIFAR-10 ResNet-20 | test accuracy: 92.0% -> 91.9% | No clear effect |
Closest papers by meaning
- On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning (AAAI 2020 · similarity 0.92)
- Decentralized Deep Learning with Arbitrary Communication Compression (ICLR 2020 · similarity 0.92)
- PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization (NeurIPS 2019 · similarity 0.91)
- Communication Compression for Decentralized Training (NeurIPS 2018 · similarity 0.90)
- Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism (NeurIPS 2025 · similarity 0.90)
- Accelerated Methods with Compressed Communications for Distributed Optimization Problems Under Data Similarity (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.