AI Paper Atlas

Research map / Optimization / Distributed and compressed

Birder: Communication-Efficient 1-bit Adaptive Optimizer for Practical Distributed DNN Training

Hanyang Peng, Shuang Qin, Yue Yu, Jin Wang, Hui Wang, Ge Li

NeurIPS 2023 · Optimization › Distributed and compressed · approach: Gradient quantization and sparsification · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use Birder instead of SGDM with 10Gbps EthernetResNet-50 DDP · ILSVRC2012, 32 samples/GPU, 64 GPUsthroughput speedup: 6189 (1.00×) -> 15253 (2.47×)Improved
Use Birder instead of BertAdam with 10Gbps EthernetBERT-Base · SQuAD 1.1, 64 GPUsthroughput speedup: 158 (1.00×) -> 990 (6.26×)Improved
Use Birder instead of 1-bit Adam, which needs 15 warmup epochsResNet-50 · ILSVRC2012, 32 samples/GPU, 64 GPUsthroughput: 5546 (0.89×) -> 15253 (2.47×)Improved
Use Birder instead of SGDM on a single node (8 GPUs)ResNet-50 · ILSVRC2012, 32 samples/GPU, 8 GPUsthroughput: 3693 (1.00×) -> 3462 (0.94×)Hurt
Use Birder instead of SGDM, small batchResNet-50 · ILSVRC2012, 32 samples/GPU, 8 GPUsTop-1 accuracy: 76.19 -> 75.98Hurt
Use Birder instead of SGDM, larger batchResNet-50 · ILSVRC2012, 128 samples/GPU, 64 GPUsTop-1 accuracy: 74.22 -> 74.24No clear effect
Use Birder instead of 1-bit Adam, larger batchResNet-50 · ILSVRC2012, 128 samples/GPU, 32 GPUsTop-1 accuracy: 74.62 -> 74.82Improved
Use Birder instead of BertAdamBERT-Base · SQuAD 1.1, 8 GPUsF1-score: 88.13 -> 88.71Improved
Use Birder instead of 1-bit AdamBERT-Base · SQuAD 1.1, 64 GPUsExact Match: 79.67 -> 80.75Improved
Replace 10Gbps Ethernet with 200Gbps InfiniBandBirder · ResNet-50, 16 GPUsspeedup over SGDM: 2.03× -> 1.45×Mixed
Replace 10Gbps Ethernet with 200Gbps InfiniBandBirder · BERT-Base SQuAD 1.1, 16 GPUsspeedup over BertAdam: 5.13× -> 2.85×Mixed
Pre-train with Birder instead of BertAdamBERT-Base · Wikipedia pre-training, 64 GPUstime-wise convergence: 4.2x fasterImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy