Research map / Optimization / Distributed and compressed
LoCo: Low-Bit Communication Adaptor for Large-Scale Model Training
Xingyu Xie, Zhijie Lin, Kim-Chuan Toh, Pan Zhou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add error feedback (ablation) | LoCo · Alpaca-GPT4 fine-tune | average accuracy: 67.4% -> 67.5% | No clear effect |
| add moving average on compression error (ablation) | LoCo · Alpaca-GPT4 fine-tune | average accuracy: 67.5% -> 67.7% | Improved |
| add 8-bit compression of stored error (ablation) | LoCo · Alpaca-GPT4 fine-tune | average accuracy: 67.8% -> 67.9% | No clear effect |
| add periodic error reset (every 512 iterations) (ablation) | LoCo · Alpaca-GPT4 fine-tune | average accuracy: 67.7% -> 67.9% | Improved |
| reset error every 128 instead of 512 iterations (ablation) | LoCo · Alpaca-GPT4 fine-tune | average accuracy: 67.9% -> 68.1% | Improved |
| replace Zero++ 4-bit with LoCo 4-bit | Zero++ · LLaMA2-7B Alpaca-GPT4 | average accuracy: 44.3% -> 45.6% | Improved |
| replace 0/1 Adam 4-bit with LoCo 4-bit | 0/1 Adam · LLaMA2-7B Alpaca-GPT4 | average accuracy: 44.8% -> 45.6% | Improved |
| use 4-bit LoCo communication instead of 16-bit Adam | Adam · LLaMA2-7B RedPajama fine-tune | average accuracy: 39.75% -> 39.70% | No clear effect |
| use 4-bit LoCo instead of PowerSGD | PowerSGD · LLaMA2-7B LoRA | average accuracy: 45.4% -> 45.7% | Improved |
| use 4-bit LoCo communication in from-scratch pre-training | Adam · Sky-MoE 300B tokens | training loss: 2.105 -> 2.102 | No clear effect |
| combine LoCo with Zero++ (LoCo-Zero++) | Zero++ · LLaMA2-0.8B and 8B pre-training | training loss | Improved |
| increase GPU count from 32 to 128 | LoCo · LLaMA2-13B Megatron-LM A800 | speedup (%): 24.58% -> 42.24% | Improved |
Closest papers by meaning
- Communication-Efficient LLM Adaptation over Decentralized GPU Meshes (NeurIPS 2026 · similarity 0.90)
- ZeRO++: Extremely Efficient Collective Communication for Large Model Training (ICLR 2024 · similarity 0.88)
- Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism (NeurIPS 2025 · similarity 0.87)
- FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training (ICML 2025 · similarity 0.87)
- MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence (NeurIPS 2024 · similarity 0.86)
- 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11) (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
