Research map / Architectures and efficiency / RNNs and normalization
Communication Efficient Distributed Training with Distributed Lion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Distributed Lion, which has workers send only sign-based binary update vectors to a server that aggregates them by averaging or majority vote, cutting communication bandwidth while matching global Lion/AdamW performance.
Problem
Distributed training of large models is bottlenecked by communicating full-precision gradients between workers and a server; the goal is an optimizer that communicates only binary or low-precision vectors while retaining Lion/AdamW-level performance.
Approach
Each worker keeps its own Lion momentum state and sends only the sign of its local Lion update; the server aggregates these binary vectors by averaging (Avg) or majority vote (MaVo) and broadcasts the result, so the update is quantized at the update level rather than the gradient level.
Key results
- On ImageNet with ViT-B/16, D-Lion (Avg) reaches 81.13 top-1 vs 80.94 for AdamW and 80.99 for G-Lion; D-Lion (MaVo) reaches 80.79.
- On OpenWebText GPT2++ 760M, D-Lion (MaVo) and G-Lion both reach 14.66 validation perplexity vs 14.70 for AdamW.
- On CIFAR-10 with 4 workers, D-Lion (MaVo) performs on par with G-Lion while outperforming GradDrop, DGC and TernGrad at matched ~1/32 bandwidth; the paper reports roughly 30-32x communication savings over global AdamW/Lion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Distributed Lion with majority-vote aggregation instead of global Lion | Lion distributed training · ImageNet-1K | Top-1 accuracy: 79.82 -> 79.69; 80.99 -> 80.79 | Hurt |
| Use Distributed Lion with majority-vote aggregation instead of global Lion | Lion distributed training · OpenWebText language modeling | perplexity: 18.35 -> 18.37; 14.66 -> 14.66 | Mixed |
| Use averaged updates instead of majority-vote aggregation (ablation) | Distributed Lion · ImageNet-1K | Top-1 accuracy: 79.69 -> 80.11; 80.79 -> 81.13 | Helped |
| Use averaged updates instead of majority-vote aggregation (ablation) | Distributed Lion · OpenWebText language modeling | perplexity: 18.37 -> 18.39; 14.66 -> 14.69 | Hurt |
| Use averaged updates instead of majority-vote aggregation (ablation) | Distributed Lion · 3-shot instruction finetuning | accuracy: 76.86 -> 76.35; 45.72 -> 45.54; 77.14 -> 76.90; 78.92 -> 78.76; 49.75 -> 48.06; 58.96 -> 59.06; 35.71 -> 32.14 | Mixed |
| Replace Lion with SIGNUM while using distributed averaging or majority vote | Distributed Lion · CIFAR-10 | testing accuracy | Hurt |
| Use Distributed Lion instead of GradDrop, DGC, or TernGrad | Distributed training · CIFAR-10 | testing accuracy | Helped |
| Increase the number of workers | Distributed training with the same optimizer · CIFAR-10 | testing accuracy | Hurt |
Surprising findings
- Averaging aggregation (D-Lion Avg) outperforms majority vote on ImageNet, while majority vote is better on language modeling and instruction finetuning, which the authors attribute to global batch size.
- D-Lion (MaVo) slightly outperforms G-Lion on CIFAR-10, which the authors hypothesize is due to reduced stochasticity from larger effective batch size.
Limitations
- Performance of Avg versus MaVo is inconsistent across datasets and benchmarks; the paper does not explain when one is preferred.
- Convergence bounds assume i.i.d. data and a bias-correction assumption; the Averaging bound does not improve with more workers.
Experimental setup
Datasets: CIFAR-10 ImageNet-1K OpenWebText
Benchmarks: CIFAR-10 ImageNet-1K OpenWebText Arc-Easy Arc-Challenge BoolQ PIQA SIQA HellaSwag OBQA
Metrics: test accuracy top-1 accuracy validation perplexity accuracy communication bits per iteration
Baselines: Global AdamW Global Lion TernGrad GradDrop Deep Gradient Compression D-SIGNUM
Builds on: Lion signSGD Signum Deep Gradient Compression TernGrad
Closest papers by meaning
- ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training (NeurIPS 2025 · similarity 0.85)
- CO2: Efficient Distributed Training with Full Communication-Computation Overlap (ICLR 2024 · similarity 0.85)
- Lion Secretly Solves a Constrained Optimization: As Lyapunov Predicts (ICLR 2024 · similarity 0.84)
- SignSGD with Federated Defense: Harnessing Adversarial Attacks through Gradient Sign Decoding (ICML 2024 · similarity 0.84)
- Convergence of Distributed Adaptive Optimization with Local Updates (ICLR 2025 · similarity 0.84)
- Occult: Optimizing Collaborative Communications across Experts for Accelerated Parallel MoE Training and Inference (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.