AI Paper Atlas

Research map / Optimization / Distributed and compressed

NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization

Ali Ramezani-Kebrya, Fartash Faghri, Ilya Markov, Vitalii Aksenov, Dan Alistarh, Daniel M. Roy

JMLR 2021 · Optimization › Distributed and compressed · approach: Gradient quantization and sparsification · 30 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use nonuniform quantization (NUQSGD 3bit) instead of ATOMO rank 3ResNet20 CIFAR-10 4 nodes · CIFAR-10accuracy: 90.5 -> 90.94Improved
use NUQSGD 3bit instead of full-precision SuperSGDResNet20 CIFAR-10 4 nodes · CIFAR-10time per step (s): 205 -> 85Improved
use NUQSGD 3bit instead of full-precision SuperSGDResNet20 CIFAR-10 4 nodes · CIFAR-10accuracy: 91.5 -> 90.94Hurt
add Huffman coding of quantized levels to NUQSGD (ablation)NUQSGD · ImageNet ResNet34/50communication (bits per component): a reduction of 67% over standard NUQSGD, approximately 2.7 bits per componentImproved
use nonuniform levels (NUQSGD) instead of QSGD uniform levels4-bit quantized SGD · CIFAR10 ResNet110 and ImageNet ResNet34test accuracyImproved
use NUQSGD instead of QSGDResNet110 CIFAR10 and ResNet18 ImageNet 8 GPUs · training losstraining lossImproved
use NUQSGD instead of QSGDinf heuristicquantized SGD · ImageNet / CIFAR10loss/accuracy/varianceNo clear effect
use NUQSGD instead of EF-SignSGD with standard hyperparametersResNet50 ImageNet 4 GPUs · ImageNetconvergence: EF-SignSGD needs bucket size 64 versus 512Improved
increase number of GPUs for DGC (versus NUQSGD) (ablation)DGC · CIFAR10 ResNet18validation accuracyHurt
quantize bias gradients (ablation)NUQSGD implementation · ImageNetaccuracyHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy