Research map / Architectures and efficiency / RNNs and normalization
Poplar: Efficient Scaling of Distributed DNN Training on Heterogeneous GPU Clusters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Poplar extends ZeRO stages 0-3 to heterogeneous GPU clusters by profiling each GPU's max batch size and speed online and searching per-GPU batch allocations, reporting 1.02-3.92x throughput over DeepSpeed and Whale.
Problem
Distributed DNN training on heterogeneous GPU clusters (differing compute, memory, and GPU counts) suffers idle time at synchronization because standard ZeRO and 3D-parallel systems assume homogeneous workers.
Approach
Treat each GPU as an independent unit within ZeRO: online profiling measures per-GPU max batch size and time-per-step per ZeRO stage, then an offline search (greedy for ZeRO-0/1, wall-time sweep over step time t for ZeRO-2/3) allocates per-GPU batch sizes with gradient accumulation, with cubic spline interpolation of speed curves and automatic ZeRO stage escalation.
Key results
- Poplar achieves 1.02-3.92x training throughput improvement over current state-of-the-art heterogeneous training systems across three heterogeneous clusters with six GPU types.
- On the 1.1B Llama and BERT comparison, Poplar reaches up to 3.92x over DeepSpeed and up to 1.98x over Whale in the third cluster.
- In ZeRO-2 and ZeRO-3, Poplar reduces gradient accumulation steps, lowering communication volume and improving performance across all three clusters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Poplar's heterogeneous-aware batch allocation and search | ZeRO training on heterogeneous GPU clusters · Three heterogeneous GPU clusters | training throughput: 1.02 ∼ 3.92x | Helped |
| Use Poplar's independent GPU allocation and profiling | ZeRO-2 and ZeRO-3 · Heterogeneous GPU clusters | cluster performance | Helped |
| Measure GPU compute capability using wall time rather than FLOPs alone | Poplar load balancing · Heterogeneous GPU clusters | compute capability measurement accuracy | Helped |
| Use a larger global batch size | Poplar · Heterogeneous GPU clusters | training performance | Helped |
| Add an A800 to the V4A3 configuration, forming V4A4 | Poplar · Cluster C, ZeRO-3 | cluster utilization | Hurt |
| Use Poplar on Cluster A with ZeRO-0 or ZeRO-1 | Poplar · Cluster A | performance gain over DeepSpeed and Whale | No clear effect |
| Use a smaller micro-batch size at 1.1B parameters | Poplar · 1.1B-parameter model | performance | Hurt |
Surprising findings
- FLOPs alone is a poor proxy for real training speed across heterogeneous GPUs; the paper reports that A100 40GB and 80GB have equal FLOPs but differ in usable max batch size and performance.
- Adding heterogeneous devices does not always improve cluster throughput; in ZeRO-3 the V4A4 group had lower utilization than V4A3 due to growing communication volume.
Limitations
- Communication bottleneck across heterogeneous network links limits scaling; adding devices can reduce throughput in ZeRO-3.
- Only a global batch size of 2 million tokens and a 0.5B Llama model were used in main experiments, with a smaller gbs than typical Llama training.
- Manually optimized baselines and the authors note Poplar gains are small in some settings (e.g., Cluster A ZeRO-0/1).
Experimental setup
Datasets: wikitext-2-v1
Benchmarks: wikitext-2-v1
Metrics: TFLOPs training throughput speedup
Baselines: DeepSpeed Whale AMP homogeneous baseline with less powerful GPUs homogeneous baseline with more powerful GPUs
Builds on: ZeRO ZeRO++ DeepSpeed
Closest papers by meaning
- Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training (NeurIPS 2026 · similarity 0.86)
- FlashMoE: Fast Distributed MoE in a Single Kernel (NeurIPS 2025 · similarity 0.86)
- Decentralized Diffusion Models (CVPR 2025 · similarity 0.86)
- From Promise to Practice: Realizing High-performance Decentralized Training (ICLR 2025 · similarity 0.85)
- Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training (ICML 2024 · similarity 0.85)
- Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.