Research map / Optimization / Distributed and compressed
Adaptive Gradient Quantization for Data-Parallel SGD
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adapt quantization levels during training with ALQ | data-parallel SGD · CIFAR-10 and ImageNet, 3 bits, 4 GPUs | validation accuracy: almost 2% on CIFAR-10 and 1% on ImageNet | Helped |
| adapt levels using ALQ rather than fixed quantization | ResNet-110 on CIFAR-10 · CIFAR-10 | validation accuracy: 93.24% ± 0.06 | Helped |
| adapt levels using ALQ rather than fixed quantization | ResNet-32 on CIFAR-10 · CIFAR-10 | validation accuracy: 91.30% ± 0.07 | Helped |
| adapt levels using ALQ rather than fixed quantization | ResNet-18 on ImageNet · ImageNet | validation accuracy: 67.72% ± 0.07 | Helped |
| use normalized ALQ instead of ALQ with norm adjustments (ablation) | ALQ · ResNet-32 on CIFAR-10 | validation accuracy: 91.30% ± 0.07; 91.96% ± 0.04 | Helped |
| use normalized ALQ instead of ALQ with norm adjustments (ablation) | ALQ · ResNet-18 on ImageNet | validation accuracy: 67.72% ± 0.07; 65.64% ± 0.07 | Hurt |
| use normalized AMQ instead of AMQ with norm adjustments (ablation) | AMQ · ResNet-110 on CIFAR-10 | validation accuracy: 92.82% ± 0.04; 92.88% ± 0.02 | No clear effect |
| use normalized AMQ instead of AMQ with norm adjustments (ablation) | AMQ · ResNet-18 on ImageNet | validation accuracy: 64.82% ± 0.05; 66.75% ± 0.05 | Helped |
| constrain adaptive levels to exponential spacing (ablation) | adaptive quantization · bucket sizes above 10^4 or below 100, and 2 bits | validation accuracy | Hurt |
| adapt quantization levels rather than use fixed levels | quantized SGD · training trajectory | quantization variance | Helped |
| use a one-time level scan rather than update levels during training (ablation) | gradient quantization · training trajectory | quantization quality | Hurt |
| increase bucket size away from 100 (ablation) | NUQSGD · ResNet-8 on CIFAR-10 | validation accuracy | Hurt |
Closest papers by meaning
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding (NeurIPS 2017 · similarity 0.91)
- Indirect Stochastic Gradient Quantization and Its Application in Distributed Deep Learning (AAAI 2020 · similarity 0.90)
- Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification and Local Computations (NeurIPS 2019 · similarity 0.88)
- Optimal and Approximate Adaptive Stochastic Quantization (NeurIPS 2024 · similarity 0.88)
- Learning from Loss Landscape: Generalizable Mixed-Precision Quantization via Adaptive Sharpness-Aware Gradient Aligning (ICML 2025 · similarity 0.86)
- Adaptive Dataset Quantization (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.