Research map / Optimization / Distributed and compressed
1-bit Adam: Communication Efficient Large-Scale Training with Adam’s Convergence Speed
Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, Yuxiong He
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use APMSqueeze instead of uncompressed BertAdam | BERT-Base and BERT-Large fine-tuning · GLUE development sets | score: RTE 68.2 -> 69.0; MRPC 84.8 -> 84.8; CoLA 56.8 -> 55.6; SST-2 91.8 -> 91.6; QNLI 90.9 -> 90.8; QQP 90.9 -> 90.9; MNLI-(m | Mixed |
| Use APMSqueeze instead of uncompressed BertAdam | BERT-Large fine-tuning · GLUE development sets | score: RTE 70.3 -> 70.4; MRPC 86.0 -> 86.1; CoLA 60.3 -> 62.0; SST-2 93.1 -> 93.8; QNLI 92.2 -> 91.9; QQP 91.4 -> 91.5; MNLI-(m | Mixed |
| Use APMSqueeze rather than Adam | BERT-Large pre-training · BERT-Large sequence length 128, 64 GPUs | training time: 174.3 hours -> 51.5 hours | Improved |
| Compress momentum with error compensation after Adam warm-up and freeze the variance term | BERT pre-training · BERT-Base and BERT-Large, both sequence length phases | communication volume: reduced into 6% of the original | Improved |
| Use APMSqueeze rather than Adam | BERT-Large pre-training · BERT-Large sequence length 128 | throughput: 3.3x | Improved |
| Use APMSqueeze rather than Adam | SQuAD fine-tuning · SQuAD 1.1 | throughput: up to 2.9x | Improved |
| Use APMSqueeze instead of Adam | BERT fine-tuning · SQuAD 1.1 | F1: 93.33 -> 93.32 | No clear effect |
| Compress gradients directly and keep updating the variance term, rather than compressing momentum and freezing variance (ablation) | Adam(1-bit Naive) · CIFAR-10 with ResNet-18 | convergence speed | Hurt |
| Use APMSqueeze with 32-bit uncompressed momentum (ablation) | APMSqueeze · CIFAR-10 with ResNet-18 | convergence speed | No clear effect |
| Replace APMSqueeze with SGD | ResNet-18 · CIFAR-10 | convergence speed | Hurt |
| Use APMSqueeze rather than uncompressed Adam | DCGAN · CelebA | training accuracy | No clear effect |
| Use APMSqueeze rather than BertAdam | BERT pre-training · BERT-Base and BERT-Large, sequence length 128 and 512 | sample-wise convergence speed | No clear effect |
Closest papers by meaning
- Adam-mini: Use Fewer Learning Rates To Gain More (ICLR 2025 · similarity 0.89)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.87)
- The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics (NeurIPS 2026 · similarity 0.87)
- SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training (ICLR 2025 · similarity 0.87)
- Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed (ICML 2025 · similarity 0.86)
- MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
