Research map / Optimization / Distributed and compressed
NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
Ali Ramezani-Kebrya, Fartash Faghri, Ilya Markov, Vitalii Aksenov, Dan Alistarh, Daniel M. Roy
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use nonuniform quantization (NUQSGD 3bit) instead of ATOMO rank 3 | ResNet20 CIFAR-10 4 nodes · CIFAR-10 | accuracy: 90.5 -> 90.94 | Improved |
| use NUQSGD 3bit instead of full-precision SuperSGD | ResNet20 CIFAR-10 4 nodes · CIFAR-10 | time per step (s): 205 -> 85 | Improved |
| use NUQSGD 3bit instead of full-precision SuperSGD | ResNet20 CIFAR-10 4 nodes · CIFAR-10 | accuracy: 91.5 -> 90.94 | Hurt |
| add Huffman coding of quantized levels to NUQSGD (ablation) | NUQSGD · ImageNet ResNet34/50 | communication (bits per component): a reduction of 67% over standard NUQSGD, approximately 2.7 bits per component | Improved |
| use nonuniform levels (NUQSGD) instead of QSGD uniform levels | 4-bit quantized SGD · CIFAR10 ResNet110 and ImageNet ResNet34 | test accuracy | Improved |
| use NUQSGD instead of QSGD | ResNet110 CIFAR10 and ResNet18 ImageNet 8 GPUs · training loss | training loss | Improved |
| use NUQSGD instead of QSGDinf heuristic | quantized SGD · ImageNet / CIFAR10 | loss/accuracy/variance | No clear effect |
| use NUQSGD instead of EF-SignSGD with standard hyperparameters | ResNet50 ImageNet 4 GPUs · ImageNet | convergence: EF-SignSGD needs bucket size 64 versus 512 | Improved |
| increase number of GPUs for DGC (versus NUQSGD) (ablation) | DGC · CIFAR10 ResNet18 | validation accuracy | Hurt |
| quantize bias gradients (ablation) | NUQSGD implementation · ImageNet | accuracy | Hurt |
Closest papers by meaning
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding (NeurIPS 2017 · similarity 0.94)
- A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication (NeurIPS 2018 · similarity 0.90)
- Quantized Distributed Training of Large Models with Convergence Guarantees (ICML 2023 · similarity 0.90)
- Adaptive Gradient Quantization for Data-Parallel SGD (NeurIPS 2020 · similarity 0.89)
- Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification and Local Computations (NeurIPS 2019 · similarity 0.89)
- Don't Compress Gradients in Random Reshuffling: Compress Gradient Differences (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
