Research map / Large language models / Instruction and fine-tuning
CocktailSGD: Fine-tuning Foundation Models over 500Mbps Networks
Jue WANG, Yucheng Lu, Binhang Yuan, Beidi Chen, Percy Liang, Christopher De Sa, Christopher Re, Ce Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove top-K sparsification from the CocktailSGD pipeline (ablation) | CocktailSGD · WikiText-103, OPT-1.3B | training loss convergence | Improved |
| remove quantization from the CocktailSGD pipeline (ablation) | CocktailSGD · WikiText-103, OPT-1.3B | training loss convergence | Improved |
| apply random sampling before top-K at the same sparsity (ablation) | top-K plus quantization · WikiText-103, OPT-1.3B | training loss convergence | No clear effect |
| apply random sampling before top-K at the same sparsity (ablation) | top-K plus quantization · WikiText-103, OPT-1.3B | communication volume | Improved |
| use top-K alone at 0.5% sparsity (ablation) | top-K sparsification · WikiText-103, OPT-1.3B | training loss convergence | Hurt |
| combine random sparsification, top-K and quantization instead of top-K or ProxSkip alone at ~100x compression | CocktailSGD · INSTRUCTIONDATA and WikiText-103 | convergence vs AllReduce: 117× vs 110× and 109× compression | Improved |
| use CocktailSGD instead of AllReduce on a 500Mbps network | GPT-NeoX-20B · Throughput under network bandwidths | training throughput: 47.2× improvement | Improved |
| use CocktailSGD instead of AllReduce on a 1Gbps network | GPT-NeoX-20B · Throughput under network bandwidths | training throughput: 30.2× improvement | Improved |
| run CocktailSGD at 500Mbps rather than AllReduce in a data center network | GPT-NeoX-20B · Throughput under network bandwidths | training throughput: 1.2× slower | Mixed |
| instruction-tune with CocktailSGD | GPT-J-6B · HELM core scenarios | average score: 7.4% | Improved |
| instruction-tune with CocktailSGD | GPT-NeoX-20B · HELM core scenarios | average score: 7.0% | Improved |
| increase single-technique compression (top-K sparsity to 0.5%, 1-bit quantization, 200 local steps) | existing compression baselines · WikiText-103, OPT-1.3B | training loss | Hurt |
Closest papers by meaning
- Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training (NeurIPS 2025 · similarity 0.89)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.88)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.87)
- HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training (ICML 2025 · similarity 0.87)
- Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes (ICML 2024 · similarity 0.87)
- BackSlash: Rate Constrained Optimized Training of Large Language Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
