Research map / Optimization / Distributed and compressed
Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 0/1 Adam instead of 1-bit Adam | BERT-Base pre-training · GLUE development set | Avg Score: 81.5 -> 81.6 | No clear effect |
| Use 0/1 Adam instead of 1-bit Adam | BERT-Large pre-training · GLUE development set | Avg Score: 83.7 -> 83.6 | No clear effect |
| Use 0/1 Adam instead of 1-bit Adam | ResNet training · ImageNet-1k validation | Top1 Acc.: 69.93 -> 69.88 | No clear effect |
| Use 0/1 Adam instead of 1-bit Adam | GPT-2 pre-training · WikiText-103 | Perplexity: 28.37 -> 28.07 | Helped |
| Use 0/1 Adam instead of 1-bit Adam | GPT-2 pre-training · LAMBADA | Acc.: 33.21 -> 33.51 | Helped |
| Combine 1-bit communication with local steps | 0/1 Adam · BERT-Base/Large, GPT-2 and ImageNet training | training throughput: up to 2× | Helped |
| Use 0/1 Adam communication instead of 1-bit Adam communication | Large-scale training · BERT-Base/Large, GPT-2 and ImageNet training | data volume: up to 87% | Helped |
| Use 0/1 Adam communication instead of 1-bit Adam communication | Large-scale training · BERT-Base/Large, GPT-2 and ImageNet training | communication rounds: 54% | Helped |
| Remove local steps while retaining the variance-freezing policy (ablation) | 0/1 Adam · Large-scale training | training throughput | Hurt |
| Remove local steps while retaining the variance-freezing policy (ablation) | 0/1 Adam · Large-scale training | data volume | Hurt |
Closest papers by meaning
- Adam-mini: Use Fewer Learning Rates To Gain More (ICLR 2025 · similarity 0.90)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.90)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.88)
- SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training (ICLR 2025 · similarity 0.88)
- Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed (ICML 2025 · similarity 0.88)
- Adam with model exponential moving average is effective for nonconvex optimization (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.