Research map / Architectures and efficiency / RNNs and normalization
AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training
Ling Chen, Houming Wu, Wenjie Yu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add zero redundancy optimizer (ablation) | AMDP · GPT-style, 8 GPUs, d=8, b=128 | throughput (ktokens/s): 80.3 -> 83.7 | Improved |
| add zero redundancy optimizer (ablation) | AMDP · BERT-style, 8 GPUs, d=8, b=128 | throughput (ktokens/s): 82.7 -> 86.1 | Improved |
| add zero redundancy optimizer (ablation) | AMDP · GPT-style, d=4 and d=8 | average GPU memory: without ZeRO memory rises by 11.4% and 19.4% | Improved |
| add zero redundancy optimizer (ablation) | AMDP · BERT-style, d=4 and d=8 | average GPU memory: without ZeRO memory rises by 9.7% and 15.4% | Improved |
| replace PipeDream-2BW with AMDP multi-directional scheduling | pipeline parallelism · GPT-style, 8 GPUs, d=8, b=128 | throughput (ktokens/s): 71.6 -> 83.7 | Improved |
| replace PipeDream-2BW with AMDP | pipeline parallelism · BERT-style, 8 GPUs, d=8, b=128 | throughput (ktokens/s): 75.8 -> 86.1 | Improved |
| replace PipeDream-2BW with AMDP on 16 GPUs | pipeline parallelism · GPT-style, 16 GPUs, d=16 | throughput (ktokens/s): 110.1 -> 124.3 | Improved |
| replace vNAG with AMDP | pipeline parallelism · Model-1 vs vNAG | throughput (ktokens/s): 47.1 -> 58.5 | Improved |
| replace vNAG with AMDP | pipeline parallelism · Model-2 vs vNAG | throughput (ktokens/s): 11.1 -> 18.2 | Improved |
| replace vNAG with AMDP | pipeline parallelism · Model-1 vs vNAG | training loss at 50k iterations: 5.12 -> 4.09 | Improved |
| replace asynchronous PipeDream-2BW with AMDP (bounded mismatch) | pipeline parallelism · GPT-style validation at 40k iterations | perplexity: 21.7 -> 2.06 | Improved |
| replace asynchronous PipeDream-2BW with AMDP (bounded mismatch) | pipeline parallelism · BERT-style validation at 40k iterations | perplexity: 65.7 -> 1.7 | Improved |
Closest papers by meaning
- AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism (NeurIPS 2026 · similarity 0.89)
- Nesterov Method for Asynchronous Pipeline Parallel Optimization (ICML 2025 · similarity 0.88)
- PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization (ICML 2025 · similarity 0.87)
- Synergistic Tensor and Pipeline Parallelism (NeurIPS 2025 · similarity 0.87)
- Pfeife: Automatic Pipeline Parallelism for PyTorch (ICML 2025 · similarity 0.86)
- Practical Performance Guarantees for Pipelined DNN Inference (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
