Research map / Architectures and efficiency / Quantization
JointSQ: Joint Sparsification-Quantization for Distributed Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JointSQ, which treats sparsification as 0-bit quantization and solves joint mixed-precision gradient compression for distributed learning via MCKP-Greedy allocation and reallocation.
Problem
Communication overhead in distributed learning; existing co-compression methods apply sparsification and quantization sequentially, yielding suboptimal solutions and architecture-specific tuning.
Approach
Treat sparsification as 0-bit quantization so that sparsification and quantization become a single mixed-precision bit allocation problem per gradient element, formulated as a multiple-choice knapsack and solved with a greedy allocation plus reallocation procedure.
Key results
- At 1000x compression JointSQ loses only 0.18% top-1 on ImageNet with ResNet-101 (vs DGC -0.44%).
- On CIFAR-100 with ResNet-18 at 32x, JointSQ improves accuracy by 1.10% over full-precision SGD (75.04 vs 73.94).
- JointSQ achieves about 1.4x to 2.9x training speedup over existing co-compression methods across 2 to 64 nodes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use JointSQ instead of QSGD | CIFAR-10, ResNet-20 at 20× compression · CIFAR-10 | accuracy: 91.56 -> 91.74 | Helped |
| Use JointSQ instead of QSGD | CIFAR-10, ResNet-110 at 20× compression · CIFAR-10 | accuracy: 93.54 -> 93.62 | Helped |
| Use JointSQ instead of QSGD | CIFAR-10, SimpleViT at 20× compression · CIFAR-10 | accuracy: 84.04 -> 84.42 | Helped |
| Use JointSQ instead of the strongest listed compressed baseline | CIFAR-10, VGG-16 at 20× compression · CIFAR-10 | accuracy: 91.41 -> 91.42 | Helped |
| Use JointSQ instead of QLSGD | CIFAR-10, ResNet-20 at 1000× compression · CIFAR-10 | accuracy: 91.21 -> 91.55 | Helped |
| Use JointSQ instead of QLSGD | CIFAR-10, ResNet-110 at 1000× compression · CIFAR-10 | accuracy: 93.17 -> 93.46 | Helped |
| Use JointSQ instead of QLSGD | CIFAR-10, VGG-16 at 1000× compression · CIFAR-10 | accuracy: 90.14 -> 90.78 | Helped |
| Use JointSQ instead of MCGQ | ResNet-18 at 32× compression · CIFAR-100 | accuracy: 74.25 -> 75.04 | Helped |
| Use JointSQ instead of MCGQ | ResNet-101 at 1000× compression · ImageNet | accuracy: 74.54 -> 74.72 | Helped |
| Use JointSQ instead of DGC | 2-LSTM at 1000× compression · PTB | perplexity: 76.85 -> 76.38 | Helped |
| Replace two-stage co-compression with joint sparsification-quantization | distributed gradient compression · theoretical compression noise analysis | compression noise | Helped |
Surprising findings
- JointSQ outperforms full-precision SGD on several settings (e.g., ResNet-18 on CIFAR-100 and 2-LSTM on PTB), which the authors attribute to compression effects.
Limitations
- Bit-width candidate set restricted to {0,2,4,8}
- Theoretical guarantee relies on bounded-variance noise proxy rather than end-to-end convergence analysis
- Dynamic compression ratio exploration is preliminary
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet PTB
Benchmarks: CIFAR-10 CIFAR-100 ImageNet PTB
Metrics: top-1 accuracy compression ratio perplexity speedup
Baselines: SGD QSGD QLSGD AC-SGD DGC MCGQ signSGD JointSQ
Builds on: QSGD DGC AC-SGD QLSGD
Closest papers by meaning
- Double Quantization for Communication-Efficient Distributed Optimization (NeurIPS 2019 · similarity 0.89)
- A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication (NeurIPS 2018 · similarity 0.88)
- Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification and Local Computations (NeurIPS 2019 · similarity 0.88)
- GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Training (NeurIPS 2018 · similarity 0.88)
- Automatic Joint Structured Pruning and Quantization for Efficient Neural Network Training and Compression (CVPR 2025 · similarity 0.87)
- Effective Interplay between Sparsity and Quantization: From Theory to Practice (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.