Research map / Architectures and efficiency / Quantization
ZeRO++: Extremely Efficient Collective Communication for Large Model Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZeRO++, three communication-reduction techniques for ZeRO-3 (quantized weight all-gather, hierarchical weight partitioning, and all-to-all based INT4 quantized gradient reduction) that cut cross-node communication volume from 3M to 0.75M and speed up training on low-bandwidth clusters.
Problem
ZeRO-3 training becomes communication-bound on low-bandwidth cross-node networks and when per-GPU batch size is small at large scale, limiting effective throughput.
Approach
Combine block-wise INT8 quantization of weights for forward all-gather, a node-level secondary FP16 weight partition that removes cross-node all-gather in backward, and a 2-hop all-to-all INT4 gradient reduction (qgZ) that dequantizes before reduction, with tensor slice reordering for correct placement and overlapped intra/inter-node pipelining.
Key results
- On a low-bandwidth (1 IB) 384-GPU cluster, ZeRO++ achieves up to 2.16x higher throughput than ZeRO-3 for 18B-138B models (e.g., 2.16x for 18B at 1K tokens/GPU).
- ZeRO++ reaches about 45% of sustained peak on high-bandwidth 384-GPU cluster, with up to 1.29x speedup over baseline.
- On GPT-350M/30B tokens, full ZeRO++ final validation loss is 2.166 vs 2.122 baseline (about 1% gap); hpZ+qwZ only matches baseline at 2.1217.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply ZeRO++ communication optimizations | ZeRO-3 · 18B, 384 GPUs, 1 IB connection, 1K tokens/GPU | throughput: 14.15 -> 30.57; 2.16x | Helped |
| apply ZeRO++ communication optimizations | ZeRO-3 · 18B, 384 GPUs, 8 IB connections, 2K tokens/GPU | throughput: 47.31 -> 53.65; 1.13x | Helped |
| enable optimized quantization kernel (ablation) | ZeRO++ · 18B, 64 GPUs | throughput: 19.73 -> 31.40 TFLOPs | Helped |
| enable fusion kernel (ablation) | ZeRO++ · 18B, 64 GPUs | throughput: 19.73 -> 21.6 TFLOPs | Helped |
| enable optimized quantization and fusion kernels (ablation) | ZeRO++ · 18B, 64 GPUs | throughput: 19.73 -> 36.16 TFLOPs | Helped |
| turn qgZ off throughout training (ablation) | ZeRO++ with hpZ and qwZ · GPT-350M, 30B tokens, Pile validation | LM loss: 2.165584 -> 2.121653 | Helped |
| use qgZ only for the first 50% of training (ablation) | ZeRO++ with hpZ and qwZ · GPT-350M, 30B tokens, Pile validation | LM loss: 2.165584 -> 2.134013 | Helped |
| use all three optimizations instead of ZeRO-3 | GPT-350M training · GPT-350M, 30B tokens, Pile validation | LM loss: 2.121762 -> 2.165584 | Hurt |
| use basic non-blocked instead of block-based quantization (ablation) | ZeRO++ · GPT-350M convergence training | convergence | Hurt |
| use hpZ and qwZ without qgZ (ablation) | ZeRO++ · GPT-350M, 30B tokens, Pile validation | LM loss: 2.121762 -> 2.121653 | No clear effect |
Surprising findings
- Naive non-blocked INT8 weight quantization diverged at the beginning of GPT-350M training, while block-based quantization converged close to baseline.
- A 2-hop (intra-node then inter-node) all-to-all is needed because a 1-hop all-to-all increases cross-node volume to N*M/Z per machine.
Limitations
- hpZ increases per-GPU memory (secondary weight copy per node), e.g. 8.9x more than ZeRO-3 for a 100B model example.
- Full qgZ on gradients incurs a small convergence gap (about 1% final loss), mitigated by enabling qgZ only for the first 50% of training.
- Evaluation limited to V100 DGX-2 nodes and GPT-style models up to 138B.
Experimental setup
Datasets: The Pile
Benchmarks: The Pile (validation LM loss)
Metrics: TFLOPs percentage of peak performance speedup validation LM loss
Baselines: ZeRO-3 MiCS PyTorch quantization non-blocked quantization
Builds on: ZeRO MiCS 1-bit Adam Megatron-Turing NLG
Closest papers by meaning
- SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training (NeurIPS 2024 · similarity 0.87)
- GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models (ICML 2025 · similarity 0.87)
- ZeCO: Zero-Communication Overhead Sequence Parallelism for Linear Attention (NeurIPS 2025 · similarity 0.86)
- Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping (ICML 2025 · similarity 0.86)
- 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11) (NeurIPS 2025 · similarity 0.86)
- Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.