Research map / Architectures and efficiency / Quantization
SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SDP4Bit, which compresses weight and gradient communication in sharded data parallelism (Megatron-LM distributed optimizer) to nearly 4 bits via quantized weight differences and two-level Hadamard-smoothed gradient quantization, with a convergence proof and up to 4.08x E2E speedup on 128 GPUs.
Problem
Communication of weights and gradients in sharded data parallelism (ShardedDP) for LLM training is a bottleneck, and existing 4-bit approaches (ZeRO++, QSDP) degrade training loss.
Approach
Quantize the difference between main weights and communicated weights rather than the weights themselves (enabling biased compressors with convergence guarantees), and use a two-level gradient scheme (INT8 intra-node, INT4 inter-node) with a fused Hadamard transform to smooth outliers.
Key results
- SDP4Bit's final validation loss is within 0.24% of the baseline across GPT-125M, 350M, and 1.3B; for GPT-6.7B the training curve closely aligns with baseline.
- Achieves up to 4.08x E2E throughput speedup for an 18B GPT model on 128 H800 GPUs (16 nodes x 8).
- Directly quantizing weights to 4 bits (qW) raises validation loss by up to 12% versus baseline, while weight-difference quantization (qWD) stays nearly identical.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SDP4Bit communication quantization rather than full-precision communication | ShardedDP training · GPT pre-training, Pile validation | validation loss: 2.29392 -> 2.29590; 2.08719 -> 2.08964; 1.92774 -> 1.93238 | No clear effect |
| Use SDP4Bit communication quantization rather than the baseline | ShardedDP training · GPT-18B, 8xH800, 16 nodes | E2E throughput (TFLOPs): 14.5 $\pm$0.07 -> 59.2 $\pm$1.37 | Helped |
| Replace direct 4-bit weight quantization with 4-bit weight-difference quantization (ablation) | SDP4Bit · GPT-125M Pile validation | validation loss: 2.57312 -> 2.29274 | Helped |
| Use direct 4-bit weight quantization instead of weight-difference quantization (ablation) | SDP4Bit · GPT-125M Pile validation | validation loss: 2.29274 -> 2.57312 | Hurt |
| Add a Hadamard smoother to two-level gradient quantization (ablation) | TLq · GPT-125M Pile validation | validation loss: 2.30479 -> 2.29528 | Helped |
| Increase TLq-HS gradient quantization group size from 128 to 512 (ablation) | SDP4Bit · GPT-125M Pile validation | validation loss: 2.29528 -> 2.29670 | Hurt |
| Use qWD group size 2048 instead of 1024 (ablation) | SDP4Bit · GPT-125M Pile validation | validation loss: 2.29338 -> 2.29274 | No clear effect |
| Fuse the Hadamard transform kernel with quantization (ablation) | SDP4Bit · GPT-1.3B over 32 A100 | E2E throughput (TFLOPs): 55.2 $\pm$0.07 -> 58.5 $\pm$0.07 | Helped |
Surprising findings
- Direct 4-bit weight quantization stays suboptimal even with group size 32, while quantizing weight differences recovers baseline loss.
Limitations
- Accuracy experiments limited to GPT models up to 6.7B parameters
- Speedup is smaller on 13B under low bandwidth due to pipeline parallelism reducing ShardedDP share of communication
- Convergence analysis relies on assumptions (L-smoothness, bounded variance) and is for SGD-style updates rather than AdamW used in experiments
Experimental setup
Datasets: The Pile
Benchmarks: The Pile (validation loss)
Metrics: validation loss E2E training throughput (TFLOPs) speedup
Baselines: Megatron-LM mixed precision baseline ZeRO++ (qW + ULq) QSDP ULq qW
Builds on: ZeRO++ QSDP Megatron-LM group-wise quantization
Closest papers by meaning
- ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs (NeurIPS 2026 · similarity 0.88)
- ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training (NeurIPS 2025 · similarity 0.88)
- Revisiting Parameter Server in LLM Post-Training (ICLR 2026 · similarity 0.88)
- ZeRO++: Extremely Efficient Collective Communication for Large Model Training (ICLR 2024 · similarity 0.87)
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.87)
- Rethinking Memory and Communication Costs for Efficient Data Parallel Training of Large Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.