Research map / Architectures and efficiency / RNNs and normalization
Memory-Efficient Pipeline-Parallel DNN Training
Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, Matei Zaharia
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Adam with 2BW's one-step-stale weight updates instead of vanilla Adam | BERT 355M · MNLI | Overall Acc.: 87.77% -> 87.82% | No clear effect |
| Use Adam with 2BW's one-step-stale weight updates instead of vanilla Adam | BERT 355M · RACE | Overall Acc.: 80.06% -> 79.48% | No clear effect |
| Use Adam with 2BW's one-step-stale weight updates instead of vanilla Adam | GPT 355M · Wikitext-103 test | perplexity: 19.28 vs. 19.56 | No clear effect |
| Use PipeDream-2BW instead of tensor model parallelism | GPT 3.8B on 64 GPUs · GPT 3.8B, 64 V100 GPUs | speedup: 20× faster | Improved |
| Use PipeDream-2BW instead of tensor model parallelism | GPT · Single node | speedup: 1.3× | Improved |
| Use pipeline parallelism with PipeDream-2BW instead of inter-layer model parallelism | GPT 2.2B · GPT 2.2B | speedup: up to 6.1× | Improved |
| Avoid periodic pipeline flushes with PipeDream-2BW instead of GPipe | GPT · GPT | throughput: up to 3.2× | Improved |
| Use 2BW instead of PipeDream-Flush | GPT · Low batch size | throughput: up to 1.6× | Improved |
| Use 2BW instead of PipeDream-Flush | GPT · Batch size 2048 | throughput: 15% | Improved |
| Use PipeDream-Flush instead of GPipe | GPT · Low batch sizes | throughput: 2.1× | Improved |
| Use PipeDream-Flush instead of GPipe | GPT · High batch sizes | throughput: 3× | Improved |
| Add activation recomputation (ablation) | PipeDream-2BW · GPT models | throughput: 33% | Mixed |
Closest papers by meaning
- Practical Performance Guarantees for Pipelined DNN Inference (ICML 2024 · similarity 0.88)
- PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization (ICML 2025 · similarity 0.88)
- Pfeife: Automatic Pipeline Parallelism for PyTorch (ICML 2025 · similarity 0.88)
- Synergistic Tensor and Pipeline Parallelism (NeurIPS 2025 · similarity 0.86)
- AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism (NeurIPS 2026 · similarity 0.86)
- Nesterov Method for Asynchronous Pipeline Parallel Optimization (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
