AI Paper Atlas

Research map / Large language models / Instruction and fine-tuning

CocktailSGD: Fine-tuning Foundation Models over 500Mbps Networks

Jue WANG, Yucheng Lu, Binhang Yuan, Beidi Chen, Percy Liang, Christopher De Sa, Christopher Re, Ce Zhang

ICML 2023 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
remove top-K sparsification from the CocktailSGD pipeline (ablation)CocktailSGD · WikiText-103, OPT-1.3Btraining loss convergenceImproved
remove quantization from the CocktailSGD pipeline (ablation)CocktailSGD · WikiText-103, OPT-1.3Btraining loss convergenceImproved
apply random sampling before top-K at the same sparsity (ablation)top-K plus quantization · WikiText-103, OPT-1.3Btraining loss convergenceNo clear effect
apply random sampling before top-K at the same sparsity (ablation)top-K plus quantization · WikiText-103, OPT-1.3Bcommunication volumeImproved
use top-K alone at 0.5% sparsity (ablation)top-K sparsification · WikiText-103, OPT-1.3Btraining loss convergenceHurt
combine random sparsification, top-K and quantization instead of top-K or ProxSkip alone at ~100x compressionCocktailSGD · INSTRUCTIONDATA and WikiText-103convergence vs AllReduce: 117× vs 110× and 109× compressionImproved
use CocktailSGD instead of AllReduce on a 500Mbps networkGPT-NeoX-20B · Throughput under network bandwidthstraining throughput: 47.2× improvementImproved
use CocktailSGD instead of AllReduce on a 1Gbps networkGPT-NeoX-20B · Throughput under network bandwidthstraining throughput: 30.2× improvementImproved
run CocktailSGD at 500Mbps rather than AllReduce in a data center networkGPT-NeoX-20B · Throughput under network bandwidthstraining throughput: 1.2× slowerMixed
instruction-tune with CocktailSGDGPT-J-6B · HELM core scenariosaverage score: 7.4%Improved
instruction-tune with CocktailSGDGPT-NeoX-20B · HELM core scenariosaverage score: 7.0%Improved
increase single-technique compression (top-K sparsity to 0.5%, 1-bit quantization, 200 local steps)existing compression baselines · WikiText-103, OPT-1.3Btraining lossHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy