Research map / Optimization / Distributed and compressed
Quantized Distributed Training of Large Models with Convergence Guarantees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Quantize transmitted weights and gradients to 8 bits with 1024-size buckets | FSDP training of GPT-125M on C4 · C4 pretraining, GPT-125M | perplexity: 35.81 -> 35.58 | Helped |
| Quantize transmitted weights and gradients to 8 bits with 1024-size buckets | FSDP training of GPT-350M on C4 · C4 pretraining, GPT-350M | perplexity: 23.94 -> 23.95 | No clear effect |
| Quantize transmitted weights and gradients to 8 bits with 1024-size buckets | FSDP training of GPT-1.3B on C4 · C4 pretraining, GPT-1.3B | perplexity: 18.00 -> 18.34 | Hurt |
| Quantize transmitted weights and gradients | FSDP training of GPT-1.3B · 10Gbps | speedup: 2.25x | Helped |
| Quantize transmitted weights and gradients | FSDP training · 100Gbps | speedup: 15% | Helped |
| Use learned rather than uniform quantization levels for W6G4 (ablation) | QSDP on GPT-125M · GPT-125M | perplexity: 35.81 -> 35.75 | Helped |
| Use learned rather than uniform quantization levels for W5G4 (ablation) | QSDP on GPT-125M · GPT-125M | perplexity: 36.34 -> 36.01 | Helped |
| Use learned rather than uniform quantization levels for W4G4 (ablation) | QSDP on GPT-125M · GPT-125M | perplexity: 37.61 -> 36.94 | Helped |
| Use learned rather than uniform quantization levels for W4G32 (ablation) | QSDP on GPT-125M · GPT-125M | perplexity: 37.11 -> 36.55 | Helped |
| Use learned quantization levels rather than uniform levels at lower bit-widths (ablation) | QSDP on GPT-125M · GPT-125M W3G32, W2G32, W8G3, and W8G2 | perplexity: 45.53 -> 42.31; 57.92 -> 56.54; 39.91 -> 37.72; 44.79 -> 44.65 | Helped |
| Reduce weight precision from 6 bits to 4 bits with uniform levels (ablation) | QSDP on GPT-125M · GPT-125M, 4-bit weights with 4-bit gradients | perplexity: 35.84 -> 37.61 | Hurt |
| Use learned quantization levels at bit-widths higher than 6 bits (ablation) | QSDP quantization · GPT-125M learned quantization experiments | perplexity | No clear effect |
Closest papers by meaning
- SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training (NeurIPS 2024 · similarity 0.90)
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding (NeurIPS 2017 · similarity 0.89)
- BitDP: Ultra-low-bit Communication for Data Parallelism in LLM Training (AAAI 2026 · similarity 0.89)
- A Quadratic Synchronization Rule for Distributed Deep Learning (ICLR 2024 · similarity 0.87)
- JointSQ: Joint Sparsification-Quantization for Distributed Learning (CVPR 2024 · similarity 0.86)
- QT-DoG: Quantization-Aware Training for Domain Generalization (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.