Research map / Architectures and efficiency / RNNs and normalization
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase AmoebaNet model capacity to 557 million parameters | AmoebaNet-B (18, 512) on ImageNet-2012 · ImageNet-2012 validation | top-1 accuracy (%): 83.9 -> 84.4 | Helped |
| Fine-tune the ImageNet-pretrained model on CIFAR-10 | AmoebaNet-B (18, 512) · CIFAR-10 | accuracy (%): 98.5 -> 99.0 | Helped |
| Fine-tune the ImageNet-pretrained model on CIFAR-100 | AmoebaNet-B (18, 512) · CIFAR-100 | accuracy (%): 89.3 -> 91.3 | Helped |
| Fine-tune the ImageNet-pretrained model on Stanford Cars | AmoebaNet-B (18, 512) · Stanford Cars | accuracy (%): 94.8 -> 94.6 | Hurt |
| Fine-tune the ImageNet-pretrained model on Oxford Pets | AmoebaNet-B (18, 512) · Oxford Pets | accuracy (%): 93.8 -> 95.9 | Helped |
| Fine-tune the ImageNet-pretrained model on Food-101 | AmoebaNet-B (18, 512) · Food-101 | accuracy (%): 90.4 -> 93.0 | Helped |
| Fine-tune the ImageNet-pretrained model on FGVC Aircraft | AmoebaNet-B (18, 512) · FGVC Aircraft | accuracy (%): 92.9 -> 92.7 | Hurt |
| Fine-tune the ImageNet-pretrained model on Birdsnap | AmoebaNet-B (18, 512) · Birdsnap | accuracy (%): 80.2 -> 83.6 | Helped |
| Increase micro-batches from M=1 to M=32 (ablation) | GPipe Transformer, K=8 · TPU training throughput | normalized training throughput: 1.3 -> 6.3 | Helped |
| Increase partitions from K=2 to K=8 with M=32 (ablation) | GPipe Transformer-48 · TPU training throughput | normalized training throughput: 1.8 -> 6.3 | Helped |
| Increase partitions from K=2 to K=8 with M=32 (ablation) | GPipe AmoebaNet-D (18, 256) · TPU training throughput | normalized training throughput: 1.21 -> 3.48 | Helped |
| Increase German-English batch size from 260K to 4M tokens (ablation) | Transformer Big · German-English validation | BLEU: 30.92 -> 32.71 | Helped |
Closest papers by meaning
- Synergistic Tensor and Pipeline Parallelism (NeurIPS 2025 · similarity 0.90)
- Pfeife: Automatic Pipeline Parallelism for PyTorch (ICML 2025 · similarity 0.88)
- Practical Performance Guarantees for Pipelined DNN Inference (ICML 2024 · similarity 0.87)
- Tensor-Parallelism with Partially Synchronized Activations (NeurIPS 2025 · similarity 0.86)
- PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization (ICML 2025 · similarity 0.86)
- A Multi-Level Framework for Accelerating Training Transformer Models (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.