Research map / Optimization / Distributed and compressed
Birder: Communication-Efficient 1-bit Adaptive Optimizer for Practical Distributed DNN Training
Hanyang Peng, Shuang Qin, Yue Yu, Jin Wang, Hui Wang, Ge Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Birder instead of SGDM with 10Gbps Ethernet | ResNet-50 DDP · ILSVRC2012, 32 samples/GPU, 64 GPUs | throughput speedup: 6189 (1.00×) -> 15253 (2.47×) | Improved |
| Use Birder instead of BertAdam with 10Gbps Ethernet | BERT-Base · SQuAD 1.1, 64 GPUs | throughput speedup: 158 (1.00×) -> 990 (6.26×) | Improved |
| Use Birder instead of 1-bit Adam, which needs 15 warmup epochs | ResNet-50 · ILSVRC2012, 32 samples/GPU, 64 GPUs | throughput: 5546 (0.89×) -> 15253 (2.47×) | Improved |
| Use Birder instead of SGDM on a single node (8 GPUs) | ResNet-50 · ILSVRC2012, 32 samples/GPU, 8 GPUs | throughput: 3693 (1.00×) -> 3462 (0.94×) | Hurt |
| Use Birder instead of SGDM, small batch | ResNet-50 · ILSVRC2012, 32 samples/GPU, 8 GPUs | Top-1 accuracy: 76.19 -> 75.98 | Hurt |
| Use Birder instead of SGDM, larger batch | ResNet-50 · ILSVRC2012, 128 samples/GPU, 64 GPUs | Top-1 accuracy: 74.22 -> 74.24 | No clear effect |
| Use Birder instead of 1-bit Adam, larger batch | ResNet-50 · ILSVRC2012, 128 samples/GPU, 32 GPUs | Top-1 accuracy: 74.62 -> 74.82 | Improved |
| Use Birder instead of BertAdam | BERT-Base · SQuAD 1.1, 8 GPUs | F1-score: 88.13 -> 88.71 | Improved |
| Use Birder instead of 1-bit Adam | BERT-Base · SQuAD 1.1, 64 GPUs | Exact Match: 79.67 -> 80.75 | Improved |
| Replace 10Gbps Ethernet with 200Gbps InfiniBand | Birder · ResNet-50, 16 GPUs | speedup over SGDM: 2.03× -> 1.45× | Mixed |
| Replace 10Gbps Ethernet with 200Gbps InfiniBand | Birder · BERT-Base SQuAD 1.1, 16 GPUs | speedup over BertAdam: 5.13× -> 2.85× | Mixed |
| Pre-train with Birder instead of BertAdam | BERT-Base · Wikipedia pre-training, 64 GPUs | time-wise convergence: 4.2x faster | Improved |
Closest papers by meaning
- Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam (ICLR 2023 · similarity 0.89)
- 1-bit Adam: Communication Efficient Large-Scale Training with Adam’s Convergence Speed (ICML 2021 · similarity 0.88)
- AdaFisher: Adaptive Second Order Optimization via Fisher Information (ICLR 2025 · similarity 0.86)
- Sketched Adaptive Distributed Deep Learning: A Sharp Convergence Analysis (NeurIPS 2025 · similarity 0.85)
- MADA: Meta-Adaptive Optimizers Through Hyper-Gradient Descent (ICML 2024 · similarity 0.84)
- Adam-mini: Use Fewer Learning Rates To Gain More (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
