Research map / Optimization / Distributed and compressed
Fine-tuning Language Models over Slow Networks using Activation Quantization with Guarantees
Jue WANG, Binhang Yuan, Luka Rimanic, Yongjun He, Tri Dao, Beidi Chen, Christopher Re, Ce Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| compress activation changes (AQ-SGD) instead of activations directly (DirectQ) (ablation) | activation quantization · QNLI/CoLA/WikiText2/arXiv, fw2-fw4 | convergence | Improved |
| use AQ-SGD instead of FP32 in slow networks | pipeline-parallel fine-tuning · GPT2-1.5B, slow networks | end-to-end speedup: 4.3× | Improved |
| AQ-SGD versus DirectQ throughput | activation quantization · GPT2-1.5B, 100 Mbps | training throughput: 3.5 -> 3.4 | No clear effect |
| AQ-SGD fw4 bw8 versus DirectQ fw4 bw8 throughput | activation quantization · GPT2-1.5B, 10 Gbps | training throughput: 4.1 -> 4.0 | No clear effect |
| quantize activations with AQ-SGD instead of FP32 | pipeline-parallel training · GPT2-1.5B, 100 Mbps | training throughput: 0.5 -> 3.4 | Improved |
| quantize activations with AQ-SGD instead of FP32 | pipeline-parallel training · GPT2-1.5B, 10 Gbps | training throughput: 3.8 -> 4.0 | No clear effect |
| reduce network bandwidth 100x (10Gbps to 100Mbps) | AQ-SGD · GPT2-1.5B | training speed: 1.18× slower | Hurt |
| combine AQ-SGD with QuantizedAdam gradient compression | end-to-end communication compression · GPT2-1.5B with data parallelism | end-to-end speedup: 4.9× | Improved |
| combine activation and gradient compression versus no compression | AQ-SGD + QuantizedAdam · GPT2-1.5B throughput | throughput: 8.5× | Improved |
| disable either activation or gradient compression (ablation) | AQ-SGD + QuantizedAdam · GPT2-1.5B throughput | throughput | Hurt |
| combine DirectQ with gradient compression | DirectQ · GPT2-1.5B | convergence | Hurt |
| lower bandwidth increases communication time | AQ-SGD fw4 bw8 · GPT2-1.5B backward comm | comm time per micro batch: 25 ms -> 125 ms | Hurt |
Closest papers by meaning
- CocktailSGD: Fine-tuning Foundation Models over 500Mbps Networks (ICML 2023 · similarity 0.89)
- On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning (AAAI 2020 · similarity 0.89)
- Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism (NeurIPS 2025 · similarity 0.88)
- SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training (NeurIPS 2024 · similarity 0.87)
- LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging (NeurIPS 2026 · similarity 0.86)
- Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
