AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Memory-Efficient Pipeline-Parallel DNN Training

Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, Matei Zaharia

ICML 2021 · Architectures and efficiency › RNNs and normalization · approach: Efficient network compression and training · 59 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use Adam with 2BW's one-step-stale weight updates instead of vanilla AdamBERT 355M · MNLIOverall Acc.: 87.77% -> 87.82%No clear effect
Use Adam with 2BW's one-step-stale weight updates instead of vanilla AdamBERT 355M · RACEOverall Acc.: 80.06% -> 79.48%No clear effect
Use Adam with 2BW's one-step-stale weight updates instead of vanilla AdamGPT 355M · Wikitext-103 testperplexity: 19.28 vs. 19.56No clear effect
Use PipeDream-2BW instead of tensor model parallelismGPT 3.8B on 64 GPUs · GPT 3.8B, 64 V100 GPUsspeedup: 20× fasterImproved
Use PipeDream-2BW instead of tensor model parallelismGPT · Single nodespeedup: 1.3×Improved
Use pipeline parallelism with PipeDream-2BW instead of inter-layer model parallelismGPT 2.2B · GPT 2.2Bspeedup: up to 6.1×Improved
Avoid periodic pipeline flushes with PipeDream-2BW instead of GPipeGPT · GPTthroughput: up to 3.2×Improved
Use 2BW instead of PipeDream-FlushGPT · Low batch sizethroughput: up to 1.6×Improved
Use 2BW instead of PipeDream-FlushGPT · Batch size 2048throughput: 15%Improved
Use PipeDream-Flush instead of GPipeGPT · Low batch sizesthroughput: 2.1×Improved
Use PipeDream-Flush instead of GPipeGPT · High batch sizesthroughput: 3×Improved
Add activation recomputation (ablation)PipeDream-2BW · GPT modelsthroughput: 33%Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy