Research map / Architectures and efficiency / RNNs and normalization
LoRDO: Distributed Low-Rank Optimization with Infrequent Communication
Andrej Jovanovic, Alex Iacob, Mher Safaryan, Ionut-Vlad Modoranu, Lorenzo Sani, William F. Shen, Xinchi Qiu, Dan Alistarh, Nicholas D. Lane
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a global projection from the aggregated pseudo-gradient without full-rank QHM (ablation) | LoRDO-Global · 16M, r=8 | final perplexity: 37.42 (local) -> 46.81 (global) | Hurt |
| use a global projection without QHM at high rank (ablation) | LoRDO-Global · 16M, r=128 | final perplexity: 35.35 (local) -> 36.65 (global) | Hurt |
| add full-rank quasi-hyperbolic momentum to global projection (ablation) | LoRDO-Global · 16M, ranks 8 to 64 | final perplexity | Improved |
| use low-rank form of QHM instead of full-rank form with global projection (ablation) | LoRDO-Global · theory and appendix A.4 | subspace stagnation | No clear effect |
| use global projection instead of local projection with QHM (ablation) | LoRDO · 16M and 125M, low ranks | final perplexity | Improved |
| reduce local batch size (more workers, fixed global batch 64) (ablation) | LoRDO-Local vs Global · 16M, r=8 | perplexity gap: Local - Global gap 1.48 to 2.30 | Hurt |
| double the global batch size (ablation) | LoRDO-Local and Global · 125M, various ranks | perplexity | Improved |
| lower synchronization frequency (larger K) (ablation) | LoRDO · 16M, r=8 vs r=128 | final perplexity | Hurt |
| add QHM term with higher beta1 under infrequent sync (ablation) | LoRDO · 16M, sync frequency sweep | final perplexity | Improved |
| use LoRDO-Global with infrequent communication vs low-rank DDP | 720M model, r=256 · 720M pretraining | perplexity: 10.34 -> 10.41 | No clear effect |
| restrict to rank 256 instead of full rank 2048 | DDP at 720M · 720M pretraining | perplexity: 9.85 -> 10.34 | Hurt |
Closest papers by meaning
- Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining (NeurIPS 2025 · similarity 0.87)
- LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics (ICLR 2025 · similarity 0.87)
- MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization (NeurIPS 2025 · similarity 0.86)
- LASER: Linear Compression in Wireless Distributed Optimization (ICML 2024 · similarity 0.86)
- From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications (ICML 2025 · similarity 0.86)
- Accelerated Methods with Compressed Communications for Distributed Optimization Problems Under Data Similarity (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
