Research map / Architectures and efficiency / Quantization
Layer-wise Quantization for Quantized Optimistic Dual Averaging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a general layer-wise unbiased quantization framework with variance and code-length bounds, and uses it in Quantized Optimistic Dual Averaging (QODA), a distributed variational-inequality solver that reduces communication and speeds up WGAN training.
Problem
Communication cost in distributed training for variational inequalities (e.g., GAN training) is a bottleneck, and existing quantization schemes are global across layers and lack rigorous guarantees for layer heterogeneity.
Approach
A layer-wise unbiased quantization framework where each layer type gets its own adaptively optimized level sequence, with tight variance and code-length bounds for general L^q normalization; combined with an optimistic dual averaging solver (QODA) that needs one gradient oracle call per iteration and adaptive learning rates without almost-sure boundedness.</222>
Key results
- QODA with layer-wise 5-bit quantization achieves up to 1.47x speedup in time per step over the uncompressed baseline at 1 Gbps bandwidth.
- Scaling to 12 GPUs yields up to 2.50x speedup per step versus the uncompressed baseline.
- On Transformer-XL with PowerSGD rank 16, layer-wise quantization gives 40.38x compression versus 27.44x for global, at similar perplexity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 5-bit QODA quantization instead of uncompressed training | WGAN training · WGAN, 1 Gbps | time per optimization step (ms): 291 -> 197 | Helped |
| Use 5-bit QODA quantization instead of uncompressed training | WGAN training · WGAN, 2.5 Gbps | time per optimization step (ms): 265 -> 195 | Helped |
| Use 5-bit QODA quantization instead of uncompressed training | WGAN training · WGAN, 5 Gbps | time per optimization step (ms): 251 -> 195 | Helped |
| Use 5-bit QODA quantization instead of uncompressed training | WGAN training · WGAN, 4 GPUs | time per optimization step (ms): 251 -> 195 | Helped |
| Use 5-bit QODA quantization instead of uncompressed training | WGAN training · WGAN, 8 GPUs | time per optimization step (ms): 303 -> 165 | Helped |
| Use 5-bit QODA quantization instead of uncompressed training | WGAN training · WGAN, 12 GPUs | time per optimization step (ms): 318 -> 127 | Helped |
| Use 5-bit QODA quantization instead of uncompressed training | WGAN training · WGAN, 16 GPUs | time per optimization step (ms): 285 -> 115 | Helped |
| Replace global quantization with layer-wise quantization | PowerSGD rank 16 on Transformer-XL · WikiText-103 test | compression rate: 27.44 -> 40.38 | Helped |
| Replace global quantization with layer-wise quantization | PowerSGD rank 32 on Transformer-XL · WikiText-103 test | compression rate: 14.07 -> 20.90 | Helped |
| Replace global quantization with layer-wise quantization | PowerSGD rank 64 on Transformer-XL · WikiText-103 test | compression rate: 7.12 -> 10.84 | Helped |
| Replace global quantization with layer-wise quantization | PowerSGD on Transformer-XL · WikiText-103 test, ranks 16, 32, and 64 | perplexity: $23.73 \pm \scriptstyle 0.16$ -> $23.70 \pm \scriptstyle 0.13$; $23.54 \pm \scriptstyle 0.13$ -> $24.08 \pm \scriptstyle | Mixed |
| Quantize only the embedding layer rather than the position-wise feed-forward or attention layer (ablation) | PowerSGD on Transformer-XL · WikiText-103 test | perplexity | Hurt |
Surprising findings
- Layer-wise compression with PowerSGD at rank 32 showed a larger perplexity standard deviation (24.08 ± 1.18), indicating instability relative to global quantization.
Limitations
- Experiments limited to WGAN on CIFAR and a single Transformer-XL setting
- Relative-noise O(1/T) rate requires co-coercivity assumption
- Coding protocol adds encoding/decoding complexity and possible retransmission overhead
Experimental setup
Datasets: CIFAR10 CIFAR100 WikiText-103
Benchmarks: CIFAR10 CIFAR100 WikiText-103
Metrics: FID test perplexity time per optimization step speedup compression rate
Baselines: Q-GenX uncompressed baseline global quantization PowerSGD
Builds on: Q-GenX L-GreCo QSGD PowerSGD Optimistic gradient
Closest papers by meaning
- Deep Neural Network Quantization via Layer-Wise Optimization Using Limited Training Data (AAAI 2019 · similarity 0.88)
- Variational Network Quantization (ICLR 2018 · similarity 0.87)
- Training Quantized Nets: A Deeper Understanding (NeurIPS 2017 · similarity 0.86)
- Gradient $\ell_1$ Regularization for Quantization Robustness (ICLR 2020 · similarity 0.86)
- Quantization without Tears (CVPR 2025 · similarity 0.85)
- LOTION: Smoothing the Optimization Landscape for Quantized Training (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.