AI Paper Atlas

Research map / Large language models / Instruction and fine-tuning

Distributed Inference and Fine-tuning of Large Language Models Over The Internet

Alexander Borzunov, Max Ryabinin, Artem Chumachenko, Dmitry Baranchuk, Tim Dettmers, Younes Belkada, Pavel Samygin, Colin Raffel

NeurIPS 2023 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 17 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use fault-tolerant dual-cache algorithm instead of caching with restartsBLOOM 7.1B inference · 128 tokens, failure rate 1e-2steps/s: 0.18 -> 3.38Improved
use fault-tolerant algorithm instead of cache-less inferenceBLOOM 7.1B inference · 1024 tokens, failure rate 1e-2steps/s: 0.89 -> 2.17Improved
use fault-tolerant algorithm instead of caching with restarts when no failures occurBLOOM 7.1B inference · 128 tokens, failure rate 0steps/s: 17.1 -> 11.4Hurt
use Petals distributed inference instead of RAM offloadingBLOOM-176B on 3xA100 · sequential inference seq len 128steps/s: 0.0495 -> 1.71Improved
use Petals distributed inference instead of RAM offloadingLlama 2 (70B) on 3xT4 · sequential inference seq len 128steps/s: 0.139 -> 2.29Improved
use Petals instead of offloading for parallel forward with slow networkBLOOM-176B on 10xRTX 3090 · parallel forward 64x128, 100 Mbit/stokens/s: 109.3 (offloading) vs 28.1Hurt
increase RTT from under 5 ms to 100 ms (ablation)Petals on Llama 2 (70B) · sequential inference seq len 128, 100 Mbit/ssteps/s: 2.29 -> 1.57Hurt
reduce bandwidth from 1 Gbit/s to 100 Mbit/s (ablation)Petals on BLOOM-176B · sequential inference seq len 128steps/s: 1.71 -> 1.66No clear effect
quantize BLOOM-176B weights to 8-bit (ablation)BLOOM-176B · zero-shot avg of three benchmarksaccuracy: 70.1 -> 70.3No clear effect
quantize OPT-175B weights to 8-bit (ablation)OPT-175B · zero-shot avg of three benchmarksaccuracy: 75.3 -> 74.9No clear effect
quantize weights to 8-bit (ablation)BLOOM-176B · generation throughput batch size 1tokens/s: 4.18 -> 3.95Hurt
use prefix tuning instead of prompt tuning (ablation)BLOOM-176B fine-tuning · forward pass, batch 8, prompt length 16tokens/s: 195.6 -> 111.0Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy